Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251214 20:52:04.296973 416 test_util.cc:276] Using random seed: -1430434604
2025-12-14T20:52:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-14T20:52:04Z Disabled control of system clock
I20251214 20:52:04.330937 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
/tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42865
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251214 20:52:04.664871 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 20:52:04.665217 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 20:52:04.665287 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 20:52:04.673571 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 20:52:04.673676 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 20:52:04.673729 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 20:52:04.673766 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 20:52:04.684036 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 14 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9516
ASAN enabled
I20251214 20:52:04.685659 424 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 20:52:04.687551 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 20:52:04.697389 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:04.697366 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:04.697446 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:04.698348 424 server_base.cc:1047] running on GCE node
I20251214 20:52:04.699276 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 20:52:04.701141 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 20:52:04.702476 424 hybrid_clock.cc:648] HybridClock initialized: now 1765745524702384 us; error 74 us; skew 500 ppm
I20251214 20:52:04.702889 424 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 20:52:04.705504 424 webserver.cc:492] Webserver started at http://127.0.104.62:44579/ using document root <none> and password file <none>
I20251214 20:52:04.706329 424 fs_manager.cc:362] Metadata directory not provided
I20251214 20:52:04.706439 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 20:52:04.706820 424 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 20:52:04.709337 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data/instance:
uuid: "c27224391721467a8f46e467a980b37a"
format_stamp: "Formatted at 2025-12-14 20:52:04 on dist-test-slave-njhz"
I20251214 20:52:04.710155 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/wal/instance:
uuid: "c27224391721467a8f46e467a980b37a"
format_stamp: "Formatted at 2025-12-14 20:52:04 on dist-test-slave-njhz"
I20251214 20:52:04.716269 424 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20251214 20:52:04.720729 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:04.722590 424 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20251214 20:52:04.722807 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/wal
uuid: "c27224391721467a8f46e467a980b37a"
format_stamp: "Formatted at 2025-12-14 20:52:04 on dist-test-slave-njhz"
I20251214 20:52:04.723032 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 20:52:04.742383 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 20:52:04.743330 424 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 20:52:04.743605 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 20:52:04.767409 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42865
I20251214 20:52:04.767470 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42865 every 8 connection(s)
I20251214 20:52:04.768965 424 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data/info.pb
I20251214 20:52:04.774279 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:04.779404 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu as pid 424
I20251214 20:52:04.779717 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/wal/instance
I20251214 20:52:04.781286 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } has no permanent_uuid. Determining permanent_uuid...
I20251214 20:52:04.782624 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
/tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:34819
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251214 20:52:04.801474 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } has no permanent_uuid. Determining permanent_uuid...
W20251214 20:52:04.802897 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111)
W20251214 20:52:04.804608 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:34819: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111)
I20251214 20:52:04.854064 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } attempt: 1
W20251214 20:52:04.856720 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:34819: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111)
I20251214 20:52:04.925072 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } attempt: 2
W20251214 20:52:04.927727 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:34819: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111)
I20251214 20:52:05.019181 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } attempt: 3
W20251214 20:52:05.021227 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:34819: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111)
W20251214 20:52:05.132668 493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 20:52:05.133013 493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 20:52:05.133625 493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 20:52:05.141937 493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 20:52:05.142047 493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 20:52:05.142119 493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 20:52:05.142181 493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 20:52:05.152678 493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:34819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 14 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9516
ASAN enabled
I20251214 20:52:05.154364 493 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 20:52:05.156294 493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251214 20:52:05.164515 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } attempt: 4
W20251214 20:52:05.167531 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:34819: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111)
W20251214 20:52:05.168314 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:05.168601 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:05.170579 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:05.171396 493 server_base.cc:1047] running on GCE node
I20251214 20:52:05.172547 493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 20:52:05.174582 493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 20:52:05.175829 493 hybrid_clock.cc:648] HybridClock initialized: now 1765745525175775 us; error 53 us; skew 500 ppm
I20251214 20:52:05.176234 493 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 20:52:05.179024 493 webserver.cc:492] Webserver started at http://127.0.104.61:34369/ using document root <none> and password file <none>
I20251214 20:52:05.180001 493 fs_manager.cc:362] Metadata directory not provided
I20251214 20:52:05.180120 493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 20:52:05.180552 493 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 20:52:05.183631 493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data/instance:
uuid: "e6713a20a82b434f8a8bc122113e26ad"
format_stamp: "Formatted at 2025-12-14 20:52:05 on dist-test-slave-njhz"
I20251214 20:52:05.184566 493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal/instance:
uuid: "e6713a20a82b434f8a8bc122113e26ad"
format_stamp: "Formatted at 2025-12-14 20:52:05 on dist-test-slave-njhz"
I20251214 20:52:05.191424 493 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20251214 20:52:05.195916 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:05.197898 493 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20251214 20:52:05.198138 493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal
uuid: "e6713a20a82b434f8a8bc122113e26ad"
format_stamp: "Formatted at 2025-12-14 20:52:05 on dist-test-slave-njhz"
I20251214 20:52:05.198400 493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 20:52:05.221073 493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 20:52:05.222186 493 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 20:52:05.222484 493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 20:52:05.248709 493 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:34819
I20251214 20:52:05.248749 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:34819 every 8 connection(s)
I20251214 20:52:05.250545 493 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data/info.pb
I20251214 20:52:05.252110 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu as pid 493
I20251214 20:52:05.252317 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal/instance
I20251214 20:52:05.253647 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
/tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:35099
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251214 20:52:05.256999 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:05.264403 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } has no permanent_uuid. Determining permanent_uuid...
I20251214 20:52:05.278923 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } has no permanent_uuid. Determining permanent_uuid...
I20251214 20:52:05.289489 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } has no permanent_uuid. Determining permanent_uuid...
W20251214 20:52:05.290798 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
W20251214 20:52:05.292721 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:35099: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:05.342134 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } attempt: 1
W20251214 20:52:05.344482 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:35099: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:05.412801 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } attempt: 2
W20251214 20:52:05.415416 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:35099: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:05.466922 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } attempt: 5
I20251214 20:52:05.475252 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } has no permanent_uuid. Determining permanent_uuid...
W20251214 20:52:05.477528 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:35099: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:05.506723 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } attempt: 3
W20251214 20:52:05.508885 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:35099: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:05.528843 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } attempt: 1
W20251214 20:52:05.531247 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:35099: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:05.599529 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } attempt: 2
W20251214 20:52:05.601752 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:35099: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
W20251214 20:52:05.612140 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 20:52:05.612475 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 20:52:05.612552 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 20:52:05.620846 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 20:52:05.620962 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 20:52:05.621016 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 20:52:05.621055 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 20:52:05.631382 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 14 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9516
ASAN enabled
I20251214 20:52:05.632979 563 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 20:52:05.634702 563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 20:52:05.644845 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:05.644630 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:05.645146 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:05.646340 563 server_base.cc:1047] running on GCE node
I20251214 20:52:05.647274 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 20:52:05.648873 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 20:52:05.650123 563 hybrid_clock.cc:648] HybridClock initialized: now 1765745525650048 us; error 67 us; skew 500 ppm
I20251214 20:52:05.650530 563 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 20:52:05.652127 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } attempt: 4
W20251214 20:52:05.654225 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:35099: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:05.658480 563 webserver.cc:492] Webserver started at http://127.0.104.60:40473/ using document root <none> and password file <none>
I20251214 20:52:05.659405 563 fs_manager.cc:362] Metadata directory not provided
I20251214 20:52:05.659533 563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 20:52:05.659914 563 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 20:52:05.662431 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data/instance:
uuid: "7c48b8eaf30c4e2caa94a51d087406d4"
format_stamp: "Formatted at 2025-12-14 20:52:05 on dist-test-slave-njhz"
I20251214 20:52:05.663336 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal/instance:
uuid: "7c48b8eaf30c4e2caa94a51d087406d4"
format_stamp: "Formatted at 2025-12-14 20:52:05 on dist-test-slave-njhz"
I20251214 20:52:05.669396 563 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251214 20:52:05.674623 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:05.676589 563 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:05.676817 563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal
uuid: "7c48b8eaf30c4e2caa94a51d087406d4"
format_stamp: "Formatted at 2025-12-14 20:52:05 on dist-test-slave-njhz"
I20251214 20:52:05.677059 563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 20:52:05.698351 563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 20:52:05.699308 563 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 20:52:05.699580 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 20:52:05.707984 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } attempt: 3
W20251214 20:52:05.710511 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:35099: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:05.722384 563 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:35099
I20251214 20:52:05.722411 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:35099 every 8 connection(s)
I20251214 20:52:05.724269 563 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data/info.pb
I20251214 20:52:05.729130 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:05.734692 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu as pid 563
I20251214 20:52:05.734928 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal/instance
I20251214 20:52:05.736150 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } has no permanent_uuid. Determining permanent_uuid...
I20251214 20:52:05.754062 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } has no permanent_uuid. Determining permanent_uuid...
I20251214 20:52:05.762470 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } has no permanent_uuid. Determining permanent_uuid...
I20251214 20:52:05.791543 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: Bootstrap starting.
I20251214 20:52:05.795441 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:05.797039 632 log.cc:826] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: Log is configured to *not* fsync() on all Append() calls
I20251214 20:52:05.800951 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: No bootstrap required, opened a new log
I20251214 20:52:05.809430 632 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:05.809928 632 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:05.810060 632 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c48b8eaf30c4e2caa94a51d087406d4, State: Initialized, Role: FOLLOWER
I20251214 20:52:05.810923 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:05.812191 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:05.812618 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:05.813633 632 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 20:52:05.821816 563 master_runner.cc:186] Error getting master registration for 127.0.104.62:42865: OK, instance_id { permanent_uuid: "c27224391721467a8f46e467a980b37a" instance_seqno: 1765745524761066 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251214 20:52:05.830207 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251214 20:52:05.830597 652 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251214 20:52:05.830731 652 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251214 20:52:05.887806 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } attempt: 4
I20251214 20:52:05.901635 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Bootstrap starting.
I20251214 20:52:05.906644 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:05.908439 490 log.cc:826] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Log is configured to *not* fsync() on all Append() calls
I20251214 20:52:05.912904 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: No bootstrap required, opened a new log
I20251214 20:52:05.918870 490 raft_consensus.cc:359] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:05.919370 490 raft_consensus.cc:385] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:05.919530 490 raft_consensus.cc:740] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c27224391721467a8f46e467a980b37a, State: Initialized, Role: FOLLOWER
I20251214 20:52:05.920228 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:05.921206 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:05.921490 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:05.922461 490 sys_catalog.cc:565] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: configured and running, proceeding with master startup.
W20251214 20:52:05.935551 666 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251214 20:52:05.935727 666 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251214 20:52:05.936656 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:34819: OK, instance_id { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" instance_seqno: 1765745525241192 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251214 20:52:05.944299 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251214 20:52:05.951960 586 tablet.cc:2378] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 20:52:05.953433 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } attempt: 5
I20251214 20:52:05.968118 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Bootstrap starting.
I20251214 20:52:05.973160 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:05.974810 561 log.cc:826] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Log is configured to *not* fsync() on all Append() calls
I20251214 20:52:05.978325 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: No bootstrap required, opened a new log
I20251214 20:52:05.986006 561 raft_consensus.cc:359] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:05.986374 561 raft_consensus.cc:385] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:05.986548 561 raft_consensus.cc:740] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6713a20a82b434f8a8bc122113e26ad, State: Initialized, Role: FOLLOWER
I20251214 20:52:05.987360 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:05.989917 561 sys_catalog.cc:565] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: configured and running, proceeding with master startup.
I20251214 20:52:05.994066 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:05.994462 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: FOLLOWER
W20251214 20:52:05.997335 444 tablet.cc:2378] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 20:52:06.011343 493 master_runner.cc:263] No leader master found from master e6713a20a82b434f8a8bc122113e26ad
I20251214 20:52:06.011488 493 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251214 20:52:06.013522 655 raft_consensus.cc:493] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:06.013783 655 raft_consensus.cc:515] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:06.015533 655 leader_election.cc:290] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:06.016343 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" is_pre_election: true
I20251214 20:52:06.016358 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
I20251214 20:52:06.016796 536 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c27224391721467a8f46e467a980b37a in term 0.
I20251214 20:52:06.016805 607 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c27224391721467a8f46e467a980b37a in term 0.
I20251214 20:52:06.017374 441 leader_election.cc:304] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, c27224391721467a8f46e467a980b37a; no voters:
I20251214 20:52:06.018090 655 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:06.018236 655 raft_consensus.cc:493] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:06.018334 655 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:06.021342 655 raft_consensus.cc:515] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:06.022357 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4"
I20251214 20:52:06.022396 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad"
I20251214 20:52:06.022550 536 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:06.022564 607 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:06.025478 536 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c27224391721467a8f46e467a980b37a in term 1.
I20251214 20:52:06.025768 607 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c27224391721467a8f46e467a980b37a in term 1.
I20251214 20:52:06.026151 442 leader_election.cc:304] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c27224391721467a8f46e467a980b37a, e6713a20a82b434f8a8bc122113e26ad; no voters:
I20251214 20:52:06.026569 655 leader_election.cc:290] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 1 election: Requested vote from peers e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:06.027873 655 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:06.028116 655 raft_consensus.cc:697] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 1 LEADER]: Becoming Leader. State: Replica: c27224391721467a8f46e467a980b37a, State: Running, Role: LEADER
I20251214 20:52:06.028905 655 consensus_queue.cc:237] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
W20251214 20:52:06.035058 680 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251214 20:52:06.035334 680 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251214 20:52:06.039285 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27224391721467a8f46e467a980b37a. Latest consensus state: current_term: 1 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:06.040007 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:06.041463 684 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 20:52:06.049755 684 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 20:52:06.070622 607 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 1 FOLLOWER]: Refusing update from remote peer c27224391721467a8f46e467a980b37a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 20:52:06.071972 655 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:06.079844 536 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 1 FOLLOWER]: Refusing update from remote peer c27224391721467a8f46e467a980b37a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 20:52:06.081043 655 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:06.116303 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27224391721467a8f46e467a980b37a. Latest consensus state: current_term: 1 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:06.116641 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:06.124614 685 mvcc.cc:204] Tried to move back new op lower bound from 7232493674768117760 to 7232493674620887040. Current Snapshot: MvccSnapshot[applied={T|T < 7232493674768117760}]
I20251214 20:52:06.126892 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27224391721467a8f46e467a980b37a. Latest consensus state: current_term: 1 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:06.127234 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:06.129838 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:06.130188 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:06.135063 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:06.136102 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:06.138931 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:06.139258 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:06.140025 684 catalog_manager.cc:1357] Generated new cluster ID: c3f04138608d4adc935bb66e1a72f2bb
I20251214 20:52:06.140144 684 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 20:52:06.144166 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:06.144452 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:06.147286 686 mvcc.cc:204] Tried to move back new op lower bound from 7232493674768117760 to 7232493674620887040. Current Snapshot: MvccSnapshot[applied={T|T < 7232493674768117760 or (T in {7232493674768117760})}]
I20251214 20:52:06.174571 684 catalog_manager.cc:1380] Generated new certificate authority record
I20251214 20:52:06.176398 684 catalog_manager.cc:1514] Loading token signing keys...
I20251214 20:52:06.201819 684 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Generated new TSK 0
I20251214 20:52:06.203677 684 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 20:52:06.229914 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
/tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--builtin_ntp_servers=127.0.104.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20251214 20:52:06.669293 694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 20:52:06.669634 694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 20:52:06.669766 694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 20:52:06.678545 694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 20:52:06.678777 694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251214 20:52:06.689718 694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 14 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9516
ASAN enabled
I20251214 20:52:06.691462 694 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 20:52:06.693271 694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251214 20:52:06.702600 702 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c27224391721467a8f46e467a980b37a)
I20251214 20:52:06.702893 702 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
W20251214 20:52:06.703646 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:06.704794 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:06.704924 703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:06.705679 702 leader_election.cc:290] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819)
I20251214 20:52:06.706048 535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
I20251214 20:52:06.706005 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a" is_pre_election: true
I20251214 20:52:06.706218 694 server_base.cc:1047] running on GCE node
I20251214 20:52:06.706936 694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 20:52:06.706951 581 leader_election.cc:304] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4; no voters: c27224391721467a8f46e467a980b37a, e6713a20a82b434f8a8bc122113e26ad
I20251214 20:52:06.707389 702 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:06.708122 694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 20:52:06.709383 694 hybrid_clock.cc:648] HybridClock initialized: now 1765745526709313 us; error 56 us; skew 500 ppm
I20251214 20:52:06.709784 694 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 20:52:06.712821 694 webserver.cc:492] Webserver started at http://127.0.104.1:33357/ using document root <none> and password file <none>
I20251214 20:52:06.713574 694 fs_manager.cc:362] Metadata directory not provided
I20251214 20:52:06.713696 694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 20:52:06.714043 694 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 20:52:06.716890 694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/data/instance:
uuid: "074399864fb2470fa9e2ee87c1fe2212"
format_stamp: "Formatted at 2025-12-14 20:52:06 on dist-test-slave-njhz"
I20251214 20:52:06.717751 694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/instance:
uuid: "074399864fb2470fa9e2ee87c1fe2212"
format_stamp: "Formatted at 2025-12-14 20:52:06 on dist-test-slave-njhz"
I20251214 20:52:06.723673 694 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20251214 20:52:06.728922 709 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:06.730825 694 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20251214 20:52:06.731050 694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal
uuid: "074399864fb2470fa9e2ee87c1fe2212"
format_stamp: "Formatted at 2025-12-14 20:52:06 on dist-test-slave-njhz"
I20251214 20:52:06.731338 694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 20:52:06.750046 694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 20:52:06.751085 694 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 20:52:06.751423 694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 20:52:06.752537 694 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 20:52:06.754565 694 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 20:52:06.754670 694 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:06.754797 694 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 20:52:06.754868 694 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:06.797973 694 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:36835
I20251214 20:52:06.798117 821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:36835 every 8 connection(s)
I20251214 20:52:06.799759 694 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/data/info.pb
I20251214 20:52:06.808440 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu as pid 694
I20251214 20:52:06.808676 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/instance
I20251214 20:52:06.817656 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
/tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--builtin_ntp_servers=127.0.104.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251214 20:52:06.832355 822 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:06.832893 822 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:06.833246 652 catalog_manager.cc:1269] Loaded cluster ID: c3f04138608d4adc935bb66e1a72f2bb
I20251214 20:52:06.833343 652 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: loading cluster ID for follower catalog manager: success
I20251214 20:52:06.834218 822 heartbeater.cc:507] Master 127.0.104.60:35099 requested a full tablet report, sending...
I20251214 20:52:06.835963 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42865
I20251214 20:52:06.836158 823 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:06.836843 823 heartbeater.cc:507] Master 127.0.104.62:42865 requested a full tablet report, sending...
I20251214 20:52:06.836999 652 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: acquiring CA information for follower catalog manager: success
I20251214 20:52:06.839371 652 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251214 20:52:06.839820 597 ts_manager.cc:194] Registered new tserver with Master: 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:06.840252 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:06.840411 824 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:06.840979 455 ts_manager.cc:194] Registered new tserver with Master: 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:06.841084 824 heartbeater.cc:507] Master 127.0.104.61:34819 requested a full tablet report, sending...
I20251214 20:52:06.842998 526 ts_manager.cc:194] Registered new tserver with Master: 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:06.843343 455 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:39607
I20251214 20:52:07.038194 680 catalog_manager.cc:1269] Loaded cluster ID: c3f04138608d4adc935bb66e1a72f2bb
I20251214 20:52:07.038322 680 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: loading cluster ID for follower catalog manager: success
I20251214 20:52:07.042140 680 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: acquiring CA information for follower catalog manager: success
I20251214 20:52:07.044305 680 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251214 20:52:07.171273 832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 20:52:07.171607 832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 20:52:07.171747 832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 20:52:07.180786 832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 20:52:07.181026 832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251214 20:52:07.182847 836 raft_consensus.cc:493] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c27224391721467a8f46e467a980b37a)
I20251214 20:52:07.183092 836 raft_consensus.cc:515] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:07.184731 836 leader_election.cc:290] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:07.185473 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a" is_pre_election: true
I20251214 20:52:07.185539 606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" is_pre_election: true
I20251214 20:52:07.186311 510 leader_election.cc:304] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6713a20a82b434f8a8bc122113e26ad; no voters: 7c48b8eaf30c4e2caa94a51d087406d4, c27224391721467a8f46e467a980b37a
I20251214 20:52:07.186740 836 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:07.192390 832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 14 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9516
ASAN enabled
I20251214 20:52:07.193976 832 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 20:52:07.195852 832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 20:52:07.208039 838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:07.208003 839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:07.208832 841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:07.209645 832 server_base.cc:1047] running on GCE node
I20251214 20:52:07.210274 832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 20:52:07.211308 832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 20:52:07.212620 832 hybrid_clock.cc:648] HybridClock initialized: now 1765745527212567 us; error 48 us; skew 500 ppm
I20251214 20:52:07.213011 832 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 20:52:07.215529 832 webserver.cc:492] Webserver started at http://127.0.104.2:39999/ using document root <none> and password file <none>
I20251214 20:52:07.216300 832 fs_manager.cc:362] Metadata directory not provided
I20251214 20:52:07.216391 832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 20:52:07.216722 832 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 20:52:07.219069 832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/data/instance:
uuid: "a2e8fd1cb731446d88f8e510c7a52999"
format_stamp: "Formatted at 2025-12-14 20:52:07 on dist-test-slave-njhz"
I20251214 20:52:07.219839 832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/instance:
uuid: "a2e8fd1cb731446d88f8e510c7a52999"
format_stamp: "Formatted at 2025-12-14 20:52:07 on dist-test-slave-njhz"
I20251214 20:52:07.225473 832 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251214 20:52:07.229418 847 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:07.231178 832 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:07.231359 832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal
uuid: "a2e8fd1cb731446d88f8e510c7a52999"
format_stamp: "Formatted at 2025-12-14 20:52:07 on dist-test-slave-njhz"
I20251214 20:52:07.231580 832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 20:52:07.250835 832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 20:52:07.251784 832 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 20:52:07.252049 832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 20:52:07.253052 832 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 20:52:07.254809 832 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 20:52:07.254897 832 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:07.254980 832 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 20:52:07.255054 832 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:07.297359 832 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:38123
I20251214 20:52:07.297448 959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:38123 every 8 connection(s)
I20251214 20:52:07.298827 832 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/data/info.pb
I20251214 20:52:07.302124 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu as pid 832
I20251214 20:52:07.302376 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/instance
I20251214 20:52:07.307929 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
/tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--builtin_ntp_servers=127.0.104.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251214 20:52:07.321130 960 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:07.321153 962 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:07.321633 962 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:07.321645 960 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:07.322706 962 heartbeater.cc:507] Master 127.0.104.61:34819 requested a full tablet report, sending...
I20251214 20:52:07.322743 960 heartbeater.cc:507] Master 127.0.104.60:35099 requested a full tablet report, sending...
I20251214 20:52:07.324862 526 ts_manager.cc:194] Registered new tserver with Master: a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:07.325500 597 ts_manager.cc:194] Registered new tserver with Master: a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:07.325757 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42865
I20251214 20:52:07.325893 961 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:07.326435 961 heartbeater.cc:507] Master 127.0.104.62:42865 requested a full tablet report, sending...
I20251214 20:52:07.327602 455 ts_manager.cc:194] Registered new tserver with Master: a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:07.328899 455 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:50727
I20251214 20:52:07.346406 823 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:07.698936 968 raft_consensus.cc:493] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c27224391721467a8f46e467a980b37a)
I20251214 20:52:07.699177 968 raft_consensus.cc:515] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:07.699855 968 leader_election.cc:290] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:07.700508 606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" is_pre_election: true
I20251214 20:52:07.700673 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a" is_pre_election: true
I20251214 20:52:07.701527 510 leader_election.cc:304] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6713a20a82b434f8a8bc122113e26ad; no voters: 7c48b8eaf30c4e2caa94a51d087406d4, c27224391721467a8f46e467a980b37a
I20251214 20:52:07.701921 968 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:07.714637 969 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c27224391721467a8f46e467a980b37a)
I20251214 20:52:07.714839 969 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:07.715536 969 leader_election.cc:290] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819)
I20251214 20:52:07.716194 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a" is_pre_election: true
I20251214 20:52:07.716289 535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
I20251214 20:52:07.716636 535 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c48b8eaf30c4e2caa94a51d087406d4 in term 1.
I20251214 20:52:07.717147 584 leader_election.cc:304] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, e6713a20a82b434f8a8bc122113e26ad; no voters: c27224391721467a8f46e467a980b37a
I20251214 20:52:07.717450 969 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 20:52:07.717563 969 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c27224391721467a8f46e467a980b37a)
I20251214 20:52:07.717670 969 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 1 FOLLOWER]: Advancing to term 2
I20251214 20:52:07.721168 969 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:07.722343 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a"
I20251214 20:52:07.723089 969 leader_election.cc:290] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 2 election: Requested vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819)
I20251214 20:52:07.723672 535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad"
I20251214 20:52:07.723965 535 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 1 FOLLOWER]: Advancing to term 2
I20251214 20:52:07.727188 535 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c48b8eaf30c4e2caa94a51d087406d4 in term 2.
I20251214 20:52:07.727679 584 leader_election.cc:304] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, e6713a20a82b434f8a8bc122113e26ad; no voters: c27224391721467a8f46e467a980b37a
I20251214 20:52:07.728137 969 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 20:52:07.728832 969 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 2 LEADER]: Becoming Leader. State: Replica: 7c48b8eaf30c4e2caa94a51d087406d4, State: Running, Role: LEADER
I20251214 20:52:07.729417 969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:07.731906 973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c48b8eaf30c4e2caa94a51d087406d4. Latest consensus state: current_term: 2 leader_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:07.732213 973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:07.733115 975 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 20:52:07.735348 975 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 20:52:07.736117 975 catalog_manager.cc:1269] Loaded cluster ID: c3f04138608d4adc935bb66e1a72f2bb
I20251214 20:52:07.736191 975 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 20:52:07.737354 975 catalog_manager.cc:1514] Loading token signing keys...
W20251214 20:52:07.737752 967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 20:52:07.738097 967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 20:52:07.738229 967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251214 20:52:07.738632 975 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: Loaded TSK: 0
I20251214 20:52:07.739727 975 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251214 20:52:07.747232 967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 20:52:07.747473 967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251214 20:52:07.758615 967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 14 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9516
ASAN enabled
I20251214 20:52:07.760751 967 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 20:52:07.762887 967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251214 20:52:07.774590 977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:07.775238 978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251214 20:52:07.776028 980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:07.777391 967 server_base.cc:1047] running on GCE node
I20251214 20:52:07.778081 967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 20:52:07.779251 967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 20:52:07.780458 967 hybrid_clock.cc:648] HybridClock initialized: now 1765745527780401 us; error 53 us; skew 500 ppm
I20251214 20:52:07.780804 967 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 20:52:07.783411 535 raft_consensus.cc:1240] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 2 FOLLOWER]: Rejecting Update request from peer c27224391721467a8f46e467a980b37a for earlier term 1. Current term is 2. Ops: []
I20251214 20:52:07.783790 967 webserver.cc:492] Webserver started at http://127.0.104.3:35435/ using document root <none> and password file <none>
I20251214 20:52:07.784128 983 consensus_queue.cc:1059] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 20:52:07.784806 986 raft_consensus.cc:3055] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 1 LEADER]: Stepping down as leader of term 1
I20251214 20:52:07.784950 986 raft_consensus.cc:740] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c27224391721467a8f46e467a980b37a, State: Running, Role: LEADER
I20251214 20:52:07.785713 967 fs_manager.cc:362] Metadata directory not provided
I20251214 20:52:07.785858 967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 20:52:07.785697 986 consensus_queue.cc:260] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:07.786383 967 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251214 20:52:07.786370 986 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 1 FOLLOWER]: Advancing to term 2
I20251214 20:52:07.790594 967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/data/instance:
uuid: "63848a9fb78a4013a6f1657621272d75"
format_stamp: "Formatted at 2025-12-14 20:52:07 on dist-test-slave-njhz"
I20251214 20:52:07.791757 967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/instance:
uuid: "63848a9fb78a4013a6f1657621272d75"
format_stamp: "Formatted at 2025-12-14 20:52:07 on dist-test-slave-njhz"
I20251214 20:52:07.799091 967 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20251214 20:52:07.804497 988 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:07.806376 967 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20251214 20:52:07.806552 967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal
uuid: "63848a9fb78a4013a6f1657621272d75"
format_stamp: "Formatted at 2025-12-14 20:52:07 on dist-test-slave-njhz"
I20251214 20:52:07.806759 967 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 20:52:07.831862 961 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:07.842970 967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 20:52:07.844193 967 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 20:52:07.844535 967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 20:52:07.845721 967 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251214 20:52:07.848009 967 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251214 20:52:07.848112 967 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:07.848210 967 ts_tablet_manager.cc:616] Registered 0 tablets
I20251214 20:52:07.848270 967 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:07.891949 967 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:33313
I20251214 20:52:07.892033 1100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:33313 every 8 connection(s)
I20251214 20:52:07.893508 967 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/data/info.pb
I20251214 20:52:07.895846 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu as pid 967
I20251214 20:52:07.896071 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/instance
I20251214 20:52:07.926077 1101 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:07.926029 1102 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42865
I20251214 20:52:07.926131 1103 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:07.926654 1101 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:07.926661 1103 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:07.926663 1102 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:07.928018 1101 heartbeater.cc:507] Master 127.0.104.60:35099 requested a full tablet report, sending...
I20251214 20:52:07.928057 1103 heartbeater.cc:507] Master 127.0.104.61:34819 requested a full tablet report, sending...
I20251214 20:52:07.928088 1102 heartbeater.cc:507] Master 127.0.104.62:42865 requested a full tablet report, sending...
I20251214 20:52:07.930048 455 ts_manager.cc:194] Registered new tserver with Master: 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:07.930100 526 ts_manager.cc:194] Registered new tserver with Master: 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:07.930195 597 ts_manager.cc:194] Registered new tserver with Master: 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:07.931977 597 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:34371
I20251214 20:52:07.938652 666 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251214 20:52:07.944710 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251214 20:52:07.992236 597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "default.Table_60948615db2346ff8f076ea71cbfa125"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:07.992632 596 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "default.Table_defedcf9bf4f4dde926055c859a26d60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:07.993238 595 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "default.Table_606bd41f7d87430bb02ffb40d27da2a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:07.993933 594 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "default.Table_1db4d218ba4c49988a4c76bbe80211d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:07.994555 597 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60948615db2346ff8f076ea71cbfa125 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:07.994753 596 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_defedcf9bf4f4dde926055c859a26d60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:07.995687 595 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_606bd41f7d87430bb02ffb40d27da2a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:07.997035 591 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "default.Table_f2aaa5a4e8264fd7908776415e23c2f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:07.997246 589 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "default.Table_b160d5f3bf5a4f8db45df3e521b43055"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:07.997430 592 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "default.Table_01557637aae04d75bde501d7b0f076fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:07.998165 589 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b160d5f3bf5a4f8db45df3e521b43055 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:07.998171 591 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2aaa5a4e8264fd7908776415e23c2f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:07.998216 592 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01557637aae04d75bde501d7b0f076fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:07.998550 590 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "default.Table_85849e4e89d243238e22ce3142f9b2ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:07.999500 593 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "default.Table_6e21ecf048a74abfa45f78f189b1b429"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:07.999606 588 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "default.Table_25a75b7ca38b49be8a79917c910689df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:08.000761 593 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e21ecf048a74abfa45f78f189b1b429 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:08.000813 588 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25a75b7ca38b49be8a79917c910689df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:08.002490 590 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85849e4e89d243238e22ce3142f9b2ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:08.003325 594 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1db4d218ba4c49988a4c76bbe80211d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:08.009748 535 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 2 FOLLOWER]: Refusing update from remote peer 7c48b8eaf30c4e2caa94a51d087406d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20251214 20:52:08.010074 465 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 2 FOLLOWER]: Refusing update from remote peer 7c48b8eaf30c4e2caa94a51d087406d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20251214 20:52:08.010717 973 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 20:52:08.011859 969 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251214 20:52:08.023620 968 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c48b8eaf30c4e2caa94a51d087406d4. Latest consensus state: current_term: 2 leader_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:08.023881 968 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:08.035351 986 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c48b8eaf30c4e2caa94a51d087406d4. Latest consensus state: current_term: 2 leader_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:08.035640 986 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:08.038439 968 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:08.038656 968 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:08.059942 986 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:08.060213 986 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:08.062315 597 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_01557637aae04d75bde501d7b0f076fb" } new_table_name: "default.Table_7aebdc494d8f44038c60435ccdc6d3fd" modify_external_catalogs: true
I20251214 20:52:08.063093 969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:08.063382 969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:08.064006 969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:08.064203 969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:08.066280 592 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_60948615db2346ff8f076ea71cbfa125" } new_table_name: "default.Table_b3c03c31d5a54b94a5f722397a259c22" modify_external_catalogs: true
I20251214 20:52:08.072311 588 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_25a75b7ca38b49be8a79917c910689df" } new_table_name: "default.Table_ac87176ddd0249279bffb87faebdab72" modify_external_catalogs: true
I20251214 20:52:08.073294 591 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_f2aaa5a4e8264fd7908776415e23c2f8" } new_table_name: "default.Table_6cedf616f8cf45efa912b2bdf51d2fb1" modify_external_catalogs: true
I20251214 20:52:08.073657 594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_defedcf9bf4f4dde926055c859a26d60" } modify_external_catalogs: true
I20251214 20:52:08.073966 594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_defedcf9bf4f4dde926055c859a26d60" } modify_external_catalogs: true
I20251214 20:52:08.075785 595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_85849e4e89d243238e22ce3142f9b2ed" } modify_external_catalogs: true
I20251214 20:52:08.075992 595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_85849e4e89d243238e22ce3142f9b2ed" } modify_external_catalogs: true
I20251214 20:52:08.077144 596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_1db4d218ba4c49988a4c76bbe80211d4" } modify_external_catalogs: true
I20251214 20:52:08.077293 596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_1db4d218ba4c49988a4c76bbe80211d4" } modify_external_catalogs: true
I20251214 20:52:08.078379 589 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_b160d5f3bf5a4f8db45df3e521b43055" } new_table_name: "default.Table_0ae641f316d547d19da925a349b1e33d" modify_external_catalogs: true
I20251214 20:52:08.079931 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu pid 563
I20251214 20:52:08.081507 627 generic_service.cc:196] Checking for leaks (request via RPC)
W20251214 20:52:08.089462 597 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e4c1451b8ea4b6f8c934633f67212b0 has no consensus state
W20251214 20:52:08.090188 592 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cefcb15849d1431fb50790343f763ba1 has no consensus state
W20251214 20:52:08.208334 584 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6e4c1451b8ea4b6f8c934633f67212b0 (table default.Table_7aebdc494d8f44038c60435ccdc6d3fd [id=a8e124ed5b7141318dfa07874e188d10], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6e4c1451b8ea4b6f8c934633f67212b0 has no consensus state
I20251214 20:52:08.081595 593 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_606bd41f7d87430bb02ffb40d27da2a9" } modify_external_catalogs: true
I20251214 20:52:08.208905 593 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_606bd41f7d87430bb02ffb40d27da2a9" } modify_external_catalogs: true
W20251214 20:52:08.209194 581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42865, user_credentials={real_user=slave}} blocked reactor thread for 118853us
I20251214 20:52:08.208741 590 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_6e21ecf048a74abfa45f78f189b1b429" } modify_external_catalogs: true
I20251214 20:52:08.209537 590 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37154:
table { table_name: "default.Table_6e21ecf048a74abfa45f78f189b1b429" } modify_external_catalogs: true
I20251214 20:52:08.211655 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu with pid 563
W20251214 20:52:08.233253 512 connection.cc:537] server connection from 127.0.0.1:48870 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251214 20:52:08.233614 1113 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.60:35099)
W20251214 20:52:08.234325 1113 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.60:35099)
W20251214 20:52:08.252858 1113 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.60:35099)
I20251214 20:52:08.329936 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
W20251214 20:52:08.331584 960 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:35099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:08.345942 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
W20251214 20:52:08.347697 822 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:35099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:08.434785 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
W20251214 20:52:08.436543 1101 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:35099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:08.589939 1170 raft_consensus.cc:493] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c48b8eaf30c4e2caa94a51d087406d4)
I20251214 20:52:08.590129 1170 raft_consensus.cc:515] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:08.590808 1170 leader_election.cc:290] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:08.591593 535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
W20251214 20:52:08.592940 441 leader_election.cc:336] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099): Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:08.593080 441 leader_election.cc:304] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c27224391721467a8f46e467a980b37a; no voters: 7c48b8eaf30c4e2caa94a51d087406d4, e6713a20a82b434f8a8bc122113e26ad
I20251214 20:52:08.593340 1170 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 20:52:08.714484 1173 raft_consensus.cc:493] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c48b8eaf30c4e2caa94a51d087406d4)
I20251214 20:52:08.714705 1173 raft_consensus.cc:515] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:08.715415 1173 leader_election.cc:290] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:08.716027 464 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a" is_pre_election: true
I20251214 20:52:08.716295 464 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6713a20a82b434f8a8bc122113e26ad in term 2.
I20251214 20:52:08.716827 510 leader_election.cc:304] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c27224391721467a8f46e467a980b37a, e6713a20a82b434f8a8bc122113e26ad; no voters:
I20251214 20:52:08.717098 1173 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251214 20:52:08.717219 1173 raft_consensus.cc:493] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7c48b8eaf30c4e2caa94a51d087406d4)
I20251214 20:52:08.717315 1173 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 2 FOLLOWER]: Advancing to term 3
W20251214 20:52:08.717648 512 leader_election.cc:336] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099): Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:08.719585 1173 raft_consensus.cc:515] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:08.720233 1173 leader_election.cc:290] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 3 election: Requested vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:08.720865 464 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a"
I20251214 20:52:08.721071 464 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 2 FOLLOWER]: Advancing to term 3
W20251214 20:52:08.721801 512 leader_election.cc:336] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099): Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111)
I20251214 20:52:08.723685 464 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6713a20a82b434f8a8bc122113e26ad in term 3.
I20251214 20:52:08.724181 510 leader_election.cc:304] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c27224391721467a8f46e467a980b37a, e6713a20a82b434f8a8bc122113e26ad; no voters: 7c48b8eaf30c4e2caa94a51d087406d4
I20251214 20:52:08.724519 1173 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 3 FOLLOWER]: Leader election won for term 3
I20251214 20:52:08.725055 1173 raft_consensus.cc:697] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 3 LEADER]: Becoming Leader. State: Replica: e6713a20a82b434f8a8bc122113e26ad, State: Running, Role: LEADER
I20251214 20:52:08.725584 1173 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:08.727349 1176 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6713a20a82b434f8a8bc122113e26ad. Latest consensus state: current_term: 3 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:08.727598 1176 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:09.083882 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
/tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:35099
--webserver_interface=127.0.104.60
--webserver_port=40473
--builtin_ntp_servers=127.0.104.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251214 20:52:09.129611 464 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 3 FOLLOWER]: Refusing update from remote peer e6713a20a82b434f8a8bc122113e26ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20251214 20:52:09.130481 1176 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20251214 20:52:09.134124 1182 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6713a20a82b434f8a8bc122113e26ad. Latest consensus state: current_term: 3 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:09.134383 1182 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:09.138474 1173 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:09.138777 1173 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:09.140174 1182 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:09.140399 1182 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:09.141566 1177 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
W20251214 20:52:09.143313 512 consensus_peers.cc:597] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad -> Peer 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099): Couldn't send request to peer 7c48b8eaf30c4e2caa94a51d087406d4. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:09.144234 1177 catalog_manager.cc:679] Loaded metadata for table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]
I20251214 20:52:09.144660 1177 catalog_manager.cc:679] Loaded metadata for table default.Table_b160d5f3bf5a4f8db45df3e521b43055 [id=2a868a8e093247baa304d4fe1422c59a]
I20251214 20:52:09.144999 1177 catalog_manager.cc:679] Loaded metadata for table default.Table_b3c03c31d5a54b94a5f722397a259c22 [id=5661f9fb5cc24abfa05d40d479b6fc1f]
I20251214 20:52:09.145280 1177 catalog_manager.cc:679] Loaded metadata for table default.Table_f2aaa5a4e8264fd7908776415e23c2f8 [id=5f062935cc214bbd8bdcbfa43b4b48f7]
I20251214 20:52:09.145548 1177 catalog_manager.cc:679] Loaded metadata for table default.Table_6e21ecf048a74abfa45f78f189b1b429 [id=8ba9cb4be2f64933b0831817ef3c1938]
I20251214 20:52:09.145864 1177 catalog_manager.cc:679] Loaded metadata for table default.Table_7aebdc494d8f44038c60435ccdc6d3fd [id=a8e124ed5b7141318dfa07874e188d10]
I20251214 20:52:09.146135 1177 catalog_manager.cc:679] Loaded metadata for table default.Table_1db4d218ba4c49988a4c76bbe80211d4 [id=ad0c020f17e349508e7855f22ad4a58b]
I20251214 20:52:09.146409 1177 catalog_manager.cc:679] Loaded metadata for table default.Table_606bd41f7d87430bb02ffb40d27da2a9 [id=e543bd053c044e5f8981a66cfc16565c]
I20251214 20:52:09.146739 1177 catalog_manager.cc:679] Loaded metadata for table default.Table_85849e4e89d243238e22ce3142f9b2ed [id=e6c1cf5557ee4cb8b7cabafbd714a92c]
I20251214 20:52:09.147066 1177 catalog_manager.cc:679] Loaded metadata for table default.Table_defedcf9bf4f4dde926055c859a26d60 [id=fff52461a49941c5a8793de581c5e205]
I20251214 20:52:09.148948 1177 tablet_loader.cc:96] loaded metadata for tablet 0a451abbc384473bb96d3ba7d6cad4b0 (table default.Table_606bd41f7d87430bb02ffb40d27da2a9 [id=e543bd053c044e5f8981a66cfc16565c])
I20251214 20:52:09.149258 1177 tablet_loader.cc:96] loaded metadata for tablet 3b28ff74ed5347238d0c5255eb2f1194 (table default.Table_defedcf9bf4f4dde926055c859a26d60 [id=fff52461a49941c5a8793de581c5e205])
I20251214 20:52:09.149513 1177 tablet_loader.cc:96] loaded metadata for tablet 4a350ae5742b40e3926a82c692933839 (table default.Table_b160d5f3bf5a4f8db45df3e521b43055 [id=2a868a8e093247baa304d4fe1422c59a])
I20251214 20:52:09.149747 1177 tablet_loader.cc:96] loaded metadata for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974])
I20251214 20:52:09.149999 1177 tablet_loader.cc:96] loaded metadata for tablet 6e4c1451b8ea4b6f8c934633f67212b0 (table default.Table_7aebdc494d8f44038c60435ccdc6d3fd [id=a8e124ed5b7141318dfa07874e188d10])
I20251214 20:52:09.150250 1177 tablet_loader.cc:96] loaded metadata for tablet 89f18e24a94e473da3f2588707c7ba8a (table default.Table_1db4d218ba4c49988a4c76bbe80211d4 [id=ad0c020f17e349508e7855f22ad4a58b])
I20251214 20:52:09.150480 1177 tablet_loader.cc:96] loaded metadata for tablet cefcb15849d1431fb50790343f763ba1 (table default.Table_b3c03c31d5a54b94a5f722397a259c22 [id=5661f9fb5cc24abfa05d40d479b6fc1f])
I20251214 20:52:09.150709 1177 tablet_loader.cc:96] loaded metadata for tablet d2abffb8700e41579f0d37e3c7fc37b1 (table default.Table_85849e4e89d243238e22ce3142f9b2ed [id=e6c1cf5557ee4cb8b7cabafbd714a92c])
I20251214 20:52:09.150944 1177 tablet_loader.cc:96] loaded metadata for tablet daf09e5c454940b9a3d4bb1812ea8b1f (table default.Table_6e21ecf048a74abfa45f78f189b1b429 [id=8ba9cb4be2f64933b0831817ef3c1938])
I20251214 20:52:09.151209 1177 tablet_loader.cc:96] loaded metadata for tablet e9e4c2715cae47a09f7117724ce34376 (table default.Table_f2aaa5a4e8264fd7908776415e23c2f8 [id=5f062935cc214bbd8bdcbfa43b4b48f7])
I20251214 20:52:09.151432 1177 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 20:52:09.152362 1177 catalog_manager.cc:1269] Loaded cluster ID: c3f04138608d4adc935bb66e1a72f2bb
I20251214 20:52:09.152447 1177 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 20:52:09.153654 1177 catalog_manager.cc:1514] Loading token signing keys...
I20251214 20:52:09.154505 1177 catalog_manager.cc:6038] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Loaded TSK: 0
I20251214 20:52:09.155426 1177 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 20:52:09.189955 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_606bd41f7d87430bb02ffb40d27da2a9" } modify_external_catalogs: true
I20251214 20:52:09.189551 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_85849e4e89d243238e22ce3142f9b2ed" } modify_external_catalogs: true
I20251214 20:52:09.190630 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_29dd5d466c7d4c1698b88311145db749"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:09.192106 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_b160d5f3bf5a4f8db45df3e521b43055" } new_table_name: "default.Table_0ae641f316d547d19da925a349b1e33d" modify_external_catalogs: true
I20251214 20:52:09.192178 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_25a75b7ca38b49be8a79917c910689df" } new_table_name: "default.Table_ac87176ddd0249279bffb87faebdab72" modify_external_catalogs: true
I20251214 20:52:09.189064 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_6e21ecf048a74abfa45f78f189b1b429" } modify_external_catalogs: true
I20251214 20:52:09.195879 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_6e21ecf048a74abfa45f78f189b1b429" } modify_external_catalogs: true
I20251214 20:52:09.191457 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_ea06b0c0cb094fba91bd4b311ba90194"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:09.196556 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_f46dedc9f5c44386abeb1d210d19b59e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:09.191803 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_defedcf9bf4f4dde926055c859a26d60" } modify_external_catalogs: true
I20251214 20:52:09.195453 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_85849e4e89d243238e22ce3142f9b2ed" } modify_external_catalogs: true
I20251214 20:52:09.197075 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_defedcf9bf4f4dde926055c859a26d60" } modify_external_catalogs: true
I20251214 20:52:09.195451 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_606bd41f7d87430bb02ffb40d27da2a9" } modify_external_catalogs: true
I20251214 20:52:09.191893 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_f2aaa5a4e8264fd7908776415e23c2f8" } new_table_name: "default.Table_6cedf616f8cf45efa912b2bdf51d2fb1" modify_external_catalogs: true
I20251214 20:52:09.191340 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1db4d218ba4c49988a4c76bbe80211d4" } modify_external_catalogs: true
I20251214 20:52:09.198720 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1db4d218ba4c49988a4c76bbe80211d4" } modify_external_catalogs: true
W20251214 20:52:09.203598 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea06b0c0cb094fba91bd4b311ba90194 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:09.203513 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f46dedc9f5c44386abeb1d210d19b59e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:09.205338 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29dd5d466c7d4c1698b88311145db749 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:09.221524 520 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a350ae5742b40e3926a82c692933839 has no consensus state
I20251214 20:52:09.233423 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_0191351f0cd945e489dc8d02dbe2c5be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.233820 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9e4c2715cae47a09f7117724ce34376 has no consensus state
W20251214 20:52:09.234411 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0191351f0cd945e489dc8d02dbe2c5be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.236099 526 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet daf09e5c454940b9a3d4bb1812ea8b1f
I20251214 20:52:09.238574 524 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 89f18e24a94e473da3f2588707c7ba8a
I20251214 20:52:09.239434 523 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet d2abffb8700e41579f0d37e3c7fc37b1
I20251214 20:52:09.239795 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_02bc643e9e7448298be1bf69cad3e9d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.240504 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02bc643e9e7448298be1bf69cad3e9d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.240989 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_bd2058c3daa94a0cbabdda39dbf5ec2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.241693 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd2058c3daa94a0cbabdda39dbf5ec2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.243012 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_8aa9003fe31c4e009bc986da7f12e868"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.243599 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8aa9003fe31c4e009bc986da7f12e868 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.244839 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_63bfde9aa135486190607e2a68c24ef4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.245333 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63bfde9aa135486190607e2a68c24ef4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:09.250813 513 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 4a350ae5742b40e3926a82c692933839 (table default.Table_0ae641f316d547d19da925a349b1e33d [id=2a868a8e093247baa304d4fe1422c59a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4a350ae5742b40e3926a82c692933839 has no consensus state
I20251214 20:52:09.279488 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_b3c03c31d5a54b94a5f722397a259c22" } new_table_name: "default.Table_af25ec2aae8a46e48066cfe3f391b504" modify_external_catalogs: true
I20251214 20:52:09.279618 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_b934001f6c8e4e7b86b4dbcbca5d066d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.280330 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b934001f6c8e4e7b86b4dbcbca5d066d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.281525 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_035a23623ce242298ab89c51024d7a09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.282100 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_035a23623ce242298ab89c51024d7a09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.289144 522 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 3b28ff74ed5347238d0c5255eb2f1194
I20251214 20:52:09.291293 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_7aebdc494d8f44038c60435ccdc6d3fd" } new_table_name: "default.Table_544d1425256741bcb7b2496d3104d9c8" modify_external_catalogs: true
I20251214 20:52:09.291746 525 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 0a451abbc384473bb96d3ba7d6cad4b0
W20251214 20:52:09.298753 521 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cefcb15849d1431fb50790343f763ba1 has no consensus state
I20251214 20:52:09.300879 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_0ae641f316d547d19da925a349b1e33d" } new_table_name: "default.Table_6d8ae2f7aafa4f28a7430e353d201b63" modify_external_catalogs: true
W20251214 20:52:09.307211 522 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e4c1451b8ea4b6f8c934633f67212b0 has no consensus state
W20251214 20:52:09.317889 525 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a350ae5742b40e3926a82c692933839 consensus state has no leader
I20251214 20:52:09.352375 755 tablet_service.cc:1505] Processing CreateTablet for tablet e9e4c2715cae47a09f7117724ce34376 (DEFAULT_TABLE table=default.Table_6cedf616f8cf45efa912b2bdf51d2fb1 [id=5f062935cc214bbd8bdcbfa43b4b48f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.354111 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e4c2715cae47a09f7117724ce34376. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.355509 753 tablet_service.cc:1505] Processing CreateTablet for tablet cefcb15849d1431fb50790343f763ba1 (DEFAULT_TABLE table=default.Table_af25ec2aae8a46e48066cfe3f391b504 [id=5661f9fb5cc24abfa05d40d479b6fc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.355983 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cefcb15849d1431fb50790343f763ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.358451 754 tablet_service.cc:1505] Processing CreateTablet for tablet 6e4c1451b8ea4b6f8c934633f67212b0 (DEFAULT_TABLE table=default.Table_544d1425256741bcb7b2496d3104d9c8 [id=a8e124ed5b7141318dfa07874e188d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.359148 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4c1451b8ea4b6f8c934633f67212b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.359970 756 tablet_service.cc:1505] Processing CreateTablet for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (DEFAULT_TABLE table=default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.360630 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cc54be0bc6d4f259967fbe0a5023f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.361500 757 tablet_service.cc:1505] Processing CreateTablet for tablet 4a350ae5742b40e3926a82c692933839 (DEFAULT_TABLE table=default.Table_6d8ae2f7aafa4f28a7430e353d201b63 [id=2a868a8e093247baa304d4fe1422c59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.361966 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a350ae5742b40e3926a82c692933839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.364174 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d (DEFAULT_TABLE table=default.Table_b934001f6c8e4e7b86b4dbcbca5d066d [id=6d4f571defdf455c8575302706c2a2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.364423 895 tablet_service.cc:1505] Processing CreateTablet for tablet 4a350ae5742b40e3926a82c692933839 (DEFAULT_TABLE table=default.Table_0ae641f316d547d19da925a349b1e33d [id=2a868a8e093247baa304d4fe1422c59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.365816 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ecf83d935c49f9b304a3355b0cc55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.366899 752 tablet_service.cc:1505] Processing CreateTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d (DEFAULT_TABLE table=default.Table_b934001f6c8e4e7b86b4dbcbca5d066d [id=6d4f571defdf455c8575302706c2a2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.367144 751 tablet_service.cc:1505] Processing CreateTablet for tablet ac19d32a31e7405b9f16566215d6b2cd (DEFAULT_TABLE table=default.Table_63bfde9aa135486190607e2a68c24ef4 [id=ac9c660a341b4fe6abedd5343196a30a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.365597 894 tablet_service.cc:1505] Processing CreateTablet for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (DEFAULT_TABLE table=default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.367729 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac19d32a31e7405b9f16566215d6b2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.367763 750 tablet_service.cc:1505] Processing CreateTablet for tablet 69d44f933f494668b2881396e02669ab (DEFAULT_TABLE table=default.Table_ea06b0c0cb094fba91bd4b311ba90194 [id=5040c9d5f4d5430dbff0ab01f2c71508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.368364 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69d44f933f494668b2881396e02669ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.368580 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cc54be0bc6d4f259967fbe0a5023f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.368628 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 47876b7d13f6455fa367933990ad2f6e (DEFAULT_TABLE table=default.Table_02bc643e9e7448298be1bf69cad3e9d0 [id=13be9c7a389b4137a6bcdf57d756bfa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.368958 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 69d44f933f494668b2881396e02669ab (DEFAULT_TABLE table=default.Table_ea06b0c0cb094fba91bd4b311ba90194 [id=5040c9d5f4d5430dbff0ab01f2c71508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.369444 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47876b7d13f6455fa367933990ad2f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.371083 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (DEFAULT_TABLE table=default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.371537 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cc54be0bc6d4f259967fbe0a5023f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.371493 749 tablet_service.cc:1505] Processing CreateTablet for tablet 47876b7d13f6455fa367933990ad2f6e (DEFAULT_TABLE table=default.Table_02bc643e9e7448298be1bf69cad3e9d0 [id=13be9c7a389b4137a6bcdf57d756bfa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.371613 748 tablet_service.cc:1505] Processing CreateTablet for tablet 7bfb575249b8403991d21c6fc96eefeb (DEFAULT_TABLE table=default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.372117 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47876b7d13f6455fa367933990ad2f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.363044 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 6e4c1451b8ea4b6f8c934633f67212b0 (DEFAULT_TABLE table=default.Table_544d1425256741bcb7b2496d3104d9c8 [id=a8e124ed5b7141318dfa07874e188d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.372584 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a350ae5742b40e3926a82c692933839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.372587 747 tablet_service.cc:1505] Processing CreateTablet for tablet 44f51e11041841e98b9a15d380226e4e (DEFAULT_TABLE table=default.Table_0191351f0cd945e489dc8d02dbe2c5be [id=3aaa67725e8442fdbae975274c8596c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.372912 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ecf83d935c49f9b304a3355b0cc55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.373157 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44f51e11041841e98b9a15d380226e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.373615 745 tablet_service.cc:1505] Processing CreateTablet for tablet 31f74b0b72b74085b31c16caca2efa08 (DEFAULT_TABLE table=default.Table_bd2058c3daa94a0cbabdda39dbf5ec2b [id=fe9038b8f14d46e8aadc6bf34505b684]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.374118 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31f74b0b72b74085b31c16caca2efa08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.374516 743 tablet_service.cc:1505] Processing CreateTablet for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (DEFAULT_TABLE table=default.Table_8aa9003fe31c4e009bc986da7f12e868 [id=0a848dd5f6474712b2eba70add3743dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.375041 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0eb9ab79d9428997c6f66790d6b1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.376117 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 6381cbf05a8c4f84b71f229497fd28ef (DEFAULT_TABLE table=default.Table_035a23623ce242298ab89c51024d7a09 [id=cd7616f6ff3244ec8628da5102e51570]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.376891 1034 tablet_service.cc:1505] Processing CreateTablet for tablet e9e4c2715cae47a09f7117724ce34376 (DEFAULT_TABLE table=default.Table_6cedf616f8cf45efa912b2bdf51d2fb1 [id=5f062935cc214bbd8bdcbfa43b4b48f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.377533 1032 tablet_service.cc:1505] Processing CreateTablet for tablet cefcb15849d1431fb50790343f763ba1 (DEFAULT_TABLE table=default.Table_af25ec2aae8a46e48066cfe3f391b504 [id=5661f9fb5cc24abfa05d40d479b6fc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.378835 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69d44f933f494668b2881396e02669ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.381754 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bfb575249b8403991d21c6fc96eefeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.383638 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 7bfb575249b8403991d21c6fc96eefeb (DEFAULT_TABLE table=default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.384338 1030 tablet_service.cc:1505] Processing CreateTablet for tablet ac19d32a31e7405b9f16566215d6b2cd (DEFAULT_TABLE table=default.Table_63bfde9aa135486190607e2a68c24ef4 [id=ac9c660a341b4fe6abedd5343196a30a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.385277 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6381cbf05a8c4f84b71f229497fd28ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.383064 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 44f51e11041841e98b9a15d380226e4e (DEFAULT_TABLE table=default.Table_0191351f0cd945e489dc8d02dbe2c5be [id=3aaa67725e8442fdbae975274c8596c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.385943 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cefcb15849d1431fb50790343f763ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.386062 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (DEFAULT_TABLE table=default.Table_8aa9003fe31c4e009bc986da7f12e868 [id=0a848dd5f6474712b2eba70add3743dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.386583 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0eb9ab79d9428997c6f66790d6b1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.387223 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 (DEFAULT_TABLE table=default.Table_f46dedc9f5c44386abeb1d210d19b59e [id=53fd8efd1a1f4b7dac3c57c8e66bdc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.387830 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e95f3d09dac4c2bb838d73f65df64a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.387298 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 4a350ae5742b40e3926a82c692933839 (DEFAULT_TABLE table=default.Table_0ae641f316d547d19da925a349b1e33d [id=2a868a8e093247baa304d4fe1422c59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.388250 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a350ae5742b40e3926a82c692933839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.390098 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44f51e11041841e98b9a15d380226e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.390895 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 31f74b0b72b74085b31c16caca2efa08 (DEFAULT_TABLE table=default.Table_bd2058c3daa94a0cbabdda39dbf5ec2b [id=fe9038b8f14d46e8aadc6bf34505b684]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.391415 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31f74b0b72b74085b31c16caca2efa08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.391798 893 tablet_service.cc:1505] Processing CreateTablet for tablet e9e4c2715cae47a09f7117724ce34376 (DEFAULT_TABLE table=default.Table_6cedf616f8cf45efa912b2bdf51d2fb1 [id=5f062935cc214bbd8bdcbfa43b4b48f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.392529 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e4c2715cae47a09f7117724ce34376. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.392545 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4c1451b8ea4b6f8c934633f67212b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.392679 744 tablet_service.cc:1505] Processing CreateTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 (DEFAULT_TABLE table=default.Table_f46dedc9f5c44386abeb1d210d19b59e [id=53fd8efd1a1f4b7dac3c57c8e66bdc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.393198 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e4c2715cae47a09f7117724ce34376. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.393127 746 tablet_service.cc:1505] Processing CreateTablet for tablet 6381cbf05a8c4f84b71f229497fd28ef (DEFAULT_TABLE table=default.Table_035a23623ce242298ab89c51024d7a09 [id=cd7616f6ff3244ec8628da5102e51570]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.393662 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e95f3d09dac4c2bb838d73f65df64a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.393849 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bfb575249b8403991d21c6fc96eefeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.394439 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac19d32a31e7405b9f16566215d6b2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.395328 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6381cbf05a8c4f84b71f229497fd28ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.405130 892 tablet_service.cc:1505] Processing CreateTablet for tablet 6e4c1451b8ea4b6f8c934633f67212b0 (DEFAULT_TABLE table=default.Table_544d1425256741bcb7b2496d3104d9c8 [id=a8e124ed5b7141318dfa07874e188d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.405897 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e4c1451b8ea4b6f8c934633f67212b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.417371 891 tablet_service.cc:1505] Processing CreateTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d (DEFAULT_TABLE table=default.Table_b934001f6c8e4e7b86b4dbcbca5d066d [id=6d4f571defdf455c8575302706c2a2b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.418248 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ecf83d935c49f9b304a3355b0cc55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.425287 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_63bfde9aa135486190607e2a68c24ef4" } new_table_name: "default.Table_86516118cb22422da5c6aeb66d0dbbc0" modify_external_catalogs: true
I20251214 20:52:09.430848 1202 tablet_bootstrap.cc:492] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:09.435050 525 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac19d32a31e7405b9f16566215d6b2cd consensus state has no leader
I20251214 20:52:09.436888 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_02bc643e9e7448298be1bf69cad3e9d0" } new_table_name: "default.Table_4a5f2b57c19d489ab805acf0e502aef3" modify_external_catalogs: true
I20251214 20:52:09.441380 889 tablet_service.cc:1505] Processing CreateTablet for tablet 7bfb575249b8403991d21c6fc96eefeb (DEFAULT_TABLE table=default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.442039 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bfb575249b8403991d21c6fc96eefeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.443077 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_bd2058c3daa94a0cbabdda39dbf5ec2b" } modify_external_catalogs: true
I20251214 20:52:09.443323 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_bd2058c3daa94a0cbabdda39dbf5ec2b" } modify_external_catalogs: true
I20251214 20:52:09.443351 890 tablet_service.cc:1505] Processing CreateTablet for tablet 47876b7d13f6455fa367933990ad2f6e (DEFAULT_TABLE table=default.Table_02bc643e9e7448298be1bf69cad3e9d0 [id=13be9c7a389b4137a6bcdf57d756bfa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.443892 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47876b7d13f6455fa367933990ad2f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.448262 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_af25ec2aae8a46e48066cfe3f391b504" } modify_external_catalogs: true
I20251214 20:52:09.448480 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_af25ec2aae8a46e48066cfe3f391b504" } modify_external_catalogs: true
I20251214 20:52:09.449971 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_b934001f6c8e4e7b86b4dbcbca5d066d" } modify_external_catalogs: true
I20251214 20:52:09.450157 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_b934001f6c8e4e7b86b4dbcbca5d066d" } modify_external_catalogs: true
I20251214 20:52:09.452672 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 31f74b0b72b74085b31c16caca2efa08
I20251214 20:52:09.456187 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_ad4bec26fc854d67b2ec37f504851a01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:09.456156 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 31f74b0b72b74085b31c16caca2efa08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:09.456831 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad4bec26fc854d67b2ec37f504851a01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.459916 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_452ec7356ff74de288f4a0fdfbc09dca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.460547 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_452ec7356ff74de288f4a0fdfbc09dca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.463861 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_4c6523df78184e2ca1544f6a3b922ccb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.464421 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c6523df78184e2ca1544f6a3b922ccb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.480131 518 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet cefcb15849d1431fb50790343f763ba1
W20251214 20:52:09.480297 525 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47876b7d13f6455fa367933990ad2f6e consensus state has no leader
I20251214 20:52:09.481563 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet cefcb15849d1431fb50790343f763ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.482018 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 23ecf83d935c49f9b304a3355b0cc55d
I20251214 20:52:09.482236 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 31f74b0b72b74085b31c16caca2efa08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.483219 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.485970 891 tablet_service.cc:1505] Processing CreateTablet for tablet 69d44f933f494668b2881396e02669ab (DEFAULT_TABLE table=default.Table_ea06b0c0cb094fba91bd4b311ba90194 [id=5040c9d5f4d5430dbff0ab01f2c71508]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.486485 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69d44f933f494668b2881396e02669ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.491349 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_6cedf616f8cf45efa912b2bdf51d2fb1" } modify_external_catalogs: true
I20251214 20:52:09.491587 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_6cedf616f8cf45efa912b2bdf51d2fb1" } modify_external_catalogs: true
I20251214 20:52:09.491721 893 tablet_service.cc:1505] Processing CreateTablet for tablet ac19d32a31e7405b9f16566215d6b2cd (DEFAULT_TABLE table=default.Table_63bfde9aa135486190607e2a68c24ef4 [id=ac9c660a341b4fe6abedd5343196a30a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.492162 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac19d32a31e7405b9f16566215d6b2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.493379 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_f46dedc9f5c44386abeb1d210d19b59e" } modify_external_catalogs: true
I20251214 20:52:09.493620 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_f46dedc9f5c44386abeb1d210d19b59e" } modify_external_catalogs: true
I20251214 20:52:09.498351 892 tablet_service.cc:1505] Processing CreateTablet for tablet cefcb15849d1431fb50790343f763ba1 (DEFAULT_TABLE table=default.Table_af25ec2aae8a46e48066cfe3f391b504 [id=5661f9fb5cc24abfa05d40d479b6fc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.498957 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cefcb15849d1431fb50790343f763ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.502223 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_86516118cb22422da5c6aeb66d0dbbc0" } new_table_name: "default.Table_f7e114a6af4742b8b0cba1d2614f2536" modify_external_catalogs: true
I20251214 20:52:09.502627 887 tablet_service.cc:1505] Processing CreateTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 (DEFAULT_TABLE table=default.Table_f46dedc9f5c44386abeb1d210d19b59e [id=53fd8efd1a1f4b7dac3c57c8e66bdc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.503278 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e95f3d09dac4c2bb838d73f65df64a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.503419 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_170ec365d121403eaaa13bb81ada4535"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:09.503547 888 tablet_service.cc:1505] Processing CreateTablet for tablet 31f74b0b72b74085b31c16caca2efa08 (DEFAULT_TABLE table=default.Table_bd2058c3daa94a0cbabdda39dbf5ec2b [id=fe9038b8f14d46e8aadc6bf34505b684]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.504062 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31f74b0b72b74085b31c16caca2efa08. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:09.504035 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_170ec365d121403eaaa13bb81ada4535 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.505956 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_eb5f20f2b40e458cade7599dbd9d5aa7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.506528 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb5f20f2b40e458cade7599dbd9d5aa7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.507917 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_a0de1527e6ca4e969729b5081dd5d147"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.508600 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0de1527e6ca4e969729b5081dd5d147 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.509012 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3c3f2d9e5c2b47ed8038a7402612b093"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.509542 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c3f2d9e5c2b47ed8038a7402612b093 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.510022 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_0d1f2c5d361d4d0d9b75b90db4433e7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.510591 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d1f2c5d361d4d0d9b75b90db4433e7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.510924 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_cf3c60b13d0542f4ae62c69c088edafb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.511477 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf3c60b13d0542f4ae62c69c088edafb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.513259 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_e23247ed295b43a1b754bf2caca42e54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.513828 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e23247ed295b43a1b754bf2caca42e54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.515828 889 tablet_service.cc:1505] Processing CreateTablet for tablet 44f51e11041841e98b9a15d380226e4e (DEFAULT_TABLE table=default.Table_0191351f0cd945e489dc8d02dbe2c5be [id=3aaa67725e8442fdbae975274c8596c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.516347 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44f51e11041841e98b9a15d380226e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.519688 890 tablet_service.cc:1505] Processing CreateTablet for tablet 6381cbf05a8c4f84b71f229497fd28ef (DEFAULT_TABLE table=default.Table_035a23623ce242298ab89c51024d7a09 [id=cd7616f6ff3244ec8628da5102e51570]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.520210 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6381cbf05a8c4f84b71f229497fd28ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.523272 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4a5f2b57c19d489ab805acf0e502aef3" } new_table_name: "default.Table_fcdbff382a58458bb3b9912ed28909bf" modify_external_catalogs: true
I20251214 20:52:09.524964 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 31f74b0b72b74085b31c16caca2efa08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.530326 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.531929 888 tablet_service.cc:1552] Processing DeleteTablet for tablet cefcb15849d1431fb50790343f763ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.535574 895 tablet_service.cc:1505] Processing CreateTablet for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (DEFAULT_TABLE table=default.Table_8aa9003fe31c4e009bc986da7f12e868 [id=0a848dd5f6474712b2eba70add3743dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.536783 741 tablet_service.cc:1552] Processing DeleteTablet for tablet cefcb15849d1431fb50790343f763ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.537377 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0eb9ab79d9428997c6f66790d6b1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.537644 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.545324 1203 tablet_bootstrap.cc:492] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:09.546449 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 31f74b0b72b74085b31c16caca2efa08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31f74b0b72b74085b31c16caca2efa08 already in progress: creating tablet
W20251214 20:52:09.547044 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 23ecf83d935c49f9b304a3355b0cc55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23ecf83d935c49f9b304a3355b0cc55d already in progress: creating tablet
W20251214 20:52:09.547688 512 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet cefcb15849d1431fb50790343f763ba1 because the tablet was not found. No further retry: Not found: Tablet not found: cefcb15849d1431fb50790343f763ba1
W20251214 20:52:09.551699 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 31f74b0b72b74085b31c16caca2efa08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31f74b0b72b74085b31c16caca2efa08 already in progress: creating tablet
W20251214 20:52:09.552467 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet cefcb15849d1431fb50790343f763ba1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cefcb15849d1431fb50790343f763ba1 already in progress: creating tablet
W20251214 20:52:09.552994 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 23ecf83d935c49f9b304a3355b0cc55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23ecf83d935c49f9b304a3355b0cc55d already in progress: creating tablet
I20251214 20:52:09.553598 1203 tablet_bootstrap.cc:654] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.555240 1203 log.cc:826] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: Log is configured to *not* fsync() on all Append() calls
I20251214 20:52:09.555243 1202 tablet_bootstrap.cc:700] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: Time spent opening tablet: real 0.122s user 0.002s sys 0.003s
I20251214 20:52:09.555872 1202 tablet_bootstrap.cc:654] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.560326 1206 tablet_bootstrap.cc:492] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:09.561928 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 31f74b0b72b74085b31c16caca2efa08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31f74b0b72b74085b31c16caca2efa08 already in progress: creating tablet
W20251214 20:52:09.562388 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet cefcb15849d1431fb50790343f763ba1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cefcb15849d1431fb50790343f763ba1 already in progress: creating tablet
W20251214 20:52:09.562630 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 23ecf83d935c49f9b304a3355b0cc55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23ecf83d935c49f9b304a3355b0cc55d already in progress: creating tablet
I20251214 20:52:09.564811 1203 tablet_bootstrap.cc:492] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:09.565130 1203 ts_tablet_manager.cc:1403] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.004s
I20251214 20:52:09.567577 1202 log.cc:826] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: Log is configured to *not* fsync() on all Append() calls
I20251214 20:52:09.569198 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_8aa9003fe31c4e009bc986da7f12e868" } new_table_name: "default.Table_3ddb3e05420d4c24abb69f51fece269a" modify_external_catalogs: true
I20251214 20:52:09.574857 1203 raft_consensus.cc:359] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.575057 756 tablet_service.cc:1505] Processing CreateTablet for tablet 35101a6997e24040994661756d9649d4 (DEFAULT_TABLE table=default.Table_452ec7356ff74de288f4a0fdfbc09dca [id=2db316c4d7814e6091fdcf30b0b4aa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.575300 1203 raft_consensus.cc:385] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.575419 1203 raft_consensus.cc:740] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.575160 895 tablet_service.cc:1505] Processing CreateTablet for tablet 35101a6997e24040994661756d9649d4 (DEFAULT_TABLE table=default.Table_452ec7356ff74de288f4a0fdfbc09dca [id=2db316c4d7814e6091fdcf30b0b4aa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.575966 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35101a6997e24040994661756d9649d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.576150 1203 consensus_queue.cc:260] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.577366 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35101a6997e24040994661756d9649d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.584616 1203 ts_tablet_manager.cc:1434] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20251214 20:52:09.585326 1202 tablet_bootstrap.cc:492] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:09.585325 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 6e4c1451b8ea4b6f8c934633f67212b0 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:09.585783 1202 ts_tablet_manager.cc:1403] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.155s user 0.011s sys 0.003s
W20251214 20:52:09.587812 522 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac19d32a31e7405b9f16566215d6b2cd consensus state has no leader
I20251214 20:52:09.588136 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 6e4c1451b8ea4b6f8c934633f67212b0 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:09.589421 1103 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:09.591442 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 35101a6997e24040994661756d9649d4 (DEFAULT_TABLE table=default.Table_452ec7356ff74de288f4a0fdfbc09dca [id=2db316c4d7814e6091fdcf30b0b4aa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.591809 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35101a6997e24040994661756d9649d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.591046 1206 tablet_bootstrap.cc:654] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.592257 742 tablet_service.cc:1505] Processing CreateTablet for tablet 0a0ca65d369a45958c6dac539964114b (DEFAULT_TABLE table=default.Table_4c6523df78184e2ca1544f6a3b922ccb [id=45ca58c541924d98bf3a7f04078ac204]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.592844 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a0ca65d369a45958c6dac539964114b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.594430 895 tablet_service.cc:1505] Processing CreateTablet for tablet 0a0ca65d369a45958c6dac539964114b (DEFAULT_TABLE table=default.Table_4c6523df78184e2ca1544f6a3b922ccb [id=45ca58c541924d98bf3a7f04078ac204]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.594655 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 0a0ca65d369a45958c6dac539964114b (DEFAULT_TABLE table=default.Table_4c6523df78184e2ca1544f6a3b922ccb [id=45ca58c541924d98bf3a7f04078ac204]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.594889 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a0ca65d369a45958c6dac539964114b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.595235 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a0ca65d369a45958c6dac539964114b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.598925 1203 tablet_bootstrap.cc:492] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:09.599623 1202 raft_consensus.cc:359] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.600956 1203 tablet_bootstrap.cc:654] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.603662 1202 raft_consensus.cc:385] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.603891 1202 raft_consensus.cc:740] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.604882 1202 consensus_queue.cc:260] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.606329 742 tablet_service.cc:1505] Processing CreateTablet for tablet ba2ed363bbab4f1f865b183297c98daa (DEFAULT_TABLE table=default.Table_ad4bec26fc854d67b2ec37f504851a01 [id=17c89c03be10405f834504b803e594c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.606775 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba2ed363bbab4f1f865b183297c98daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.607676 1203 tablet_bootstrap.cc:492] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:09.607798 1203 ts_tablet_manager.cc:1403] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20251214 20:52:09.608085 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 1e95f3d09dac4c2bb838d73f65df64a6
I20251214 20:52:09.609014 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.609365 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:09.609954 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 1e95f3d09dac4c2bb838d73f65df64a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e95f3d09dac4c2bb838d73f65df64a6 already in progress: creating tablet
W20251214 20:52:09.610141 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 1e95f3d09dac4c2bb838d73f65df64a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e95f3d09dac4c2bb838d73f65df64a6 already in progress: creating tablet
I20251214 20:52:09.610460 1203 raft_consensus.cc:359] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.610648 1203 raft_consensus.cc:385] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.610749 1203 raft_consensus.cc:740] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.611008 1203 consensus_queue.cc:260] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.611840 1202 ts_tablet_manager.cc:1434] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.026s user 0.013s sys 0.001s
I20251214 20:52:09.614261 1203 ts_tablet_manager.cc:1434] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251214 20:52:09.615690 1021 tablet_service.cc:1505] Processing CreateTablet for tablet ba2ed363bbab4f1f865b183297c98daa (DEFAULT_TABLE table=default.Table_ad4bec26fc854d67b2ec37f504851a01 [id=17c89c03be10405f834504b803e594c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.616173 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba2ed363bbab4f1f865b183297c98daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.616379 1203 tablet_bootstrap.cc:492] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:09.618287 1203 tablet_bootstrap.cc:654] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.619159 895 tablet_service.cc:1505] Processing CreateTablet for tablet ba2ed363bbab4f1f865b183297c98daa (DEFAULT_TABLE table=default.Table_ad4bec26fc854d67b2ec37f504851a01 [id=17c89c03be10405f834504b803e594c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.619702 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba2ed363bbab4f1f865b183297c98daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.620512 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 6e4c1451b8ea4b6f8c934633f67212b0 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:09.621022 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 6e4c1451b8ea4b6f8c934633f67212b0 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:09.622241 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
W20251214 20:52:09.623070 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 1e95f3d09dac4c2bb838d73f65df64a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e95f3d09dac4c2bb838d73f65df64a6 already in progress: creating tablet
I20251214 20:52:09.622383 1206 log.cc:826] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: Log is configured to *not* fsync() on all Append() calls
I20251214 20:52:09.625844 962 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:09.627686 1202 tablet_bootstrap.cc:492] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:09.632206 1202 tablet_bootstrap.cc:654] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.639621 1202 tablet_bootstrap.cc:492] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:09.639959 1202 ts_tablet_manager.cc:1403] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20251214 20:52:09.640388 1206 tablet_bootstrap.cc:492] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:09.641479 1202 raft_consensus.cc:359] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.641904 1206 ts_tablet_manager.cc:1403] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.004s
I20251214 20:52:09.642086 1202 raft_consensus.cc:385] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.642236 1202 raft_consensus.cc:740] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.642603 1202 consensus_queue.cc:260] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.643249 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 6e4c1451b8ea4b6f8c934633f67212b0 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:09.646654 1202 ts_tablet_manager.cc:1434] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:09.647574 1203 tablet_bootstrap.cc:492] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:09.647820 1203 ts_tablet_manager.cc:1403] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.017s
I20251214 20:52:09.648731 1203 raft_consensus.cc:359] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:09.648810 1104 tablet.cc:2378] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 20:52:09.650027 1203 raft_consensus.cc:385] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.650235 1203 raft_consensus.cc:740] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.650627 1202 tablet_bootstrap.cc:492] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:09.650625 1203 consensus_queue.cc:260] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.650918 1206 raft_consensus.cc:359] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.651432 1206 raft_consensus.cc:385] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.651554 1206 raft_consensus.cc:740] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.652500 1206 consensus_queue.cc:260] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.652649 1202 tablet_bootstrap.cc:654] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.654615 1203 ts_tablet_manager.cc:1434] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:09.659397 1203 tablet_bootstrap.cc:492] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:09.661351 1203 tablet_bootstrap.cc:654] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.663204 1206 ts_tablet_manager.cc:1434] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.020s user 0.010s sys 0.008s
I20251214 20:52:09.664011 824 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:09.667757 1206 tablet_bootstrap.cc:492] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:09.670126 1206 tablet_bootstrap.cc:654] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.674774 1202 tablet_bootstrap.cc:492] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:09.674976 1202 ts_tablet_manager.cc:1403] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.004s
I20251214 20:52:09.675566 1203 tablet_bootstrap.cc:492] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:09.675747 1203 ts_tablet_manager.cc:1403] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20251214 20:52:09.676045 1202 raft_consensus.cc:359] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.676306 1202 raft_consensus.cc:385] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.676390 1202 raft_consensus.cc:740] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.676757 1203 raft_consensus.cc:359] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.676954 1203 raft_consensus.cc:385] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.677058 1203 raft_consensus.cc:740] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.677316 1203 consensus_queue.cc:260] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.677875 1206 tablet_bootstrap.cc:492] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:09.678143 1206 ts_tablet_manager.cc:1403] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20251214 20:52:09.679178 1202 consensus_queue.cc:260] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.679773 1206 raft_consensus.cc:359] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.680039 1206 raft_consensus.cc:385] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.680172 1206 raft_consensus.cc:740] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.680497 1206 consensus_queue.cc:260] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.680814 1203 ts_tablet_manager.cc:1434] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:09.682183 1206 ts_tablet_manager.cc:1434] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:09.683460 1206 tablet_bootstrap.cc:492] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:09.685082 1203 tablet_bootstrap.cc:492] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:09.685382 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 6e4c1451b8ea4b6f8c934633f67212b0
I20251214 20:52:09.686169 517 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 4a350ae5742b40e3926a82c692933839 (table default.Table_6d8ae2f7aafa4f28a7430e353d201b63 [id=2a868a8e093247baa304d4fe1422c59a]). Expected version 2 got 0
I20251214 20:52:09.686446 517 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:09.686487 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e4c1451b8ea4b6f8c934633f67212b0 with delete_type TABLET_DATA_DELETED (replaced by tablet d8947ebdc87341ad8b1a01f5dd246172) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.686815 517 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ac19d32a31e7405b9f16566215d6b2cd (table default.Table_f7e114a6af4742b8b0cba1d2614f2536 [id=ac9c660a341b4fe6abedd5343196a30a]). Expected version 2 got 0
I20251214 20:52:09.687011 517 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 47876b7d13f6455fa367933990ad2f6e (table default.Table_4a5f2b57c19d489ab805acf0e502aef3 [id=13be9c7a389b4137a6bcdf57d756bfa7]). Expected version 1 got 0
W20251214 20:52:09.687546 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 6e4c1451b8ea4b6f8c934633f67212b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e4c1451b8ea4b6f8c934633f67212b0 already in progress: creating tablet
I20251214 20:52:09.688557 519 catalog_manager.cc:6724] ReplaceTablet: tablet 6e4c1451b8ea4b6f8c934633f67212b0 deleted and replaced by tablet d8947ebdc87341ad8b1a01f5dd246172
I20251214 20:52:09.688983 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a0de1527e6ca4e969729b5081dd5d147" } new_table_name: "default.Table_ab99f8e2fad843e6966b1d86e41d58f6" modify_external_catalogs: true
I20251214 20:52:09.690249 1202 ts_tablet_manager.cc:1434] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20251214 20:52:09.691335 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e4c1451b8ea4b6f8c934633f67212b0 with delete_type TABLET_DATA_DELETED (replaced by tablet d8947ebdc87341ad8b1a01f5dd246172) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.691720 1207 tablet_replica.cc:333] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:09.692283 1203 tablet_bootstrap.cc:654] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.692317 1207 raft_consensus.cc:2243] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:09.692786 1207 raft_consensus.cc:2272] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.694430 1202 tablet_bootstrap.cc:492] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:09.694518 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 358d5c6b594147c8ba136b4b0732fae1 has no consensus state
I20251214 20:52:09.695588 1207 ts_tablet_manager.cc:1916] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:09.697332 1202 tablet_bootstrap.cc:654] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.697324 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_6d8ae2f7aafa4f28a7430e353d201b63" } modify_external_catalogs: true
I20251214 20:52:09.697561 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_6d8ae2f7aafa4f28a7430e353d201b63" } modify_external_catalogs: true
I20251214 20:52:09.700886 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e4c1451b8ea4b6f8c934633f67212b0 with delete_type TABLET_DATA_DELETED (replaced by tablet d8947ebdc87341ad8b1a01f5dd246172) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.701293 1208 tablet_replica.cc:333] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:09.701903 1208 raft_consensus.cc:2243] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:09.702436 1208 raft_consensus.cc:2272] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.704603 1208 ts_tablet_manager.cc:1916] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:09.705044 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_f7e114a6af4742b8b0cba1d2614f2536" } new_table_name: "default.Table_8090a0016b2248f6b8fa395e4f676734" modify_external_catalogs: true
I20251214 20:52:09.706662 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_eb5f20f2b40e458cade7599dbd9d5aa7" } modify_external_catalogs: true
I20251214 20:52:09.706823 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_eb5f20f2b40e458cade7599dbd9d5aa7" } modify_external_catalogs: true
I20251214 20:52:09.707872 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_92c09424acc6404ea18b929d27d505b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.708539 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92c09424acc6404ea18b929d27d505b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.711532 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_d7cbf1ffbce44772951752ecbb001da6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.712165 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7cbf1ffbce44772951752ecbb001da6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.713222 1208 ts_tablet_manager.cc:1929] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:09.713413 1208 log.cc:1199] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/6e4c1451b8ea4b6f8c934633f67212b0
I20251214 20:52:09.713892 1208 ts_tablet_manager.cc:1950] T 6e4c1451b8ea4b6f8c934633f67212b0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:09.714823 523 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 47876b7d13f6455fa367933990ad2f6e (table default.Table_4a5f2b57c19d489ab805acf0e502aef3 [id=13be9c7a389b4137a6bcdf57d756bfa7]). Expected version 1 got 0
I20251214 20:52:09.715409 523 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:09.715698 523 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet ac19d32a31e7405b9f16566215d6b2cd (table default.Table_f7e114a6af4742b8b0cba1d2614f2536 [id=ac9c660a341b4fe6abedd5343196a30a]). Expected version 2 got 0
I20251214 20:52:09.716116 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 6e4c1451b8ea4b6f8c934633f67212b0 (table default.Table_544d1425256741bcb7b2496d3104d9c8 [id=a8e124ed5b7141318dfa07874e188d10]) successfully deleted
W20251214 20:52:09.716306 524 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a0eb9ab79d9428997c6f66790d6b1a3 consensus state has no leader
W20251214 20:52:09.717288 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47876b7d13f6455fa367933990ad2f6e consensus state has no leader
I20251214 20:52:09.717729 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3f3f4981717d406895af06c998f950d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.718467 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f3f4981717d406895af06c998f950d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.720122 524 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet ac19d32a31e7405b9f16566215d6b2cd (table default.Table_f7e114a6af4742b8b0cba1d2614f2536 [id=ac9c660a341b4fe6abedd5343196a30a]). Expected version 2 got 0
I20251214 20:52:09.720510 524 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:09.720882 1202 tablet_bootstrap.cc:492] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:09.720953 524 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (table default.Table_3ddb3e05420d4c24abb69f51fece269a [id=0a848dd5f6474712b2eba70add3743dc]). Expected version 1 got 0
I20251214 20:52:09.721071 1202 ts_tablet_manager.cc:1403] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20251214 20:52:09.721182 524 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 47876b7d13f6455fa367933990ad2f6e (table default.Table_fcdbff382a58458bb3b9912ed28909bf [id=13be9c7a389b4137a6bcdf57d756bfa7]). Expected version 2 got 0
I20251214 20:52:09.722033 1202 raft_consensus.cc:359] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.722278 1202 raft_consensus.cc:385] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.722350 1202 raft_consensus.cc:740] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.722645 1202 consensus_queue.cc:260] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.724467 756 tablet_service.cc:1505] Processing CreateTablet for tablet a05f1ec28fb24508ae937c70aaaefffe (DEFAULT_TABLE table=default.Table_0d1f2c5d361d4d0d9b75b90db4433e7b [id=69175aeba7894328af1e0fe73b55025d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.725212 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 4a350ae5742b40e3926a82c692933839
I20251214 20:52:09.726366 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a350ae5742b40e3926a82c692933839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.726594 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet e9e4c2715cae47a09f7117724ce34376
I20251214 20:52:09.726804 895 tablet_service.cc:1505] Processing CreateTablet for tablet a05f1ec28fb24508ae937c70aaaefffe (DEFAULT_TABLE table=default.Table_0d1f2c5d361d4d0d9b75b90db4433e7b [id=69175aeba7894328af1e0fe73b55025d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:09.727140 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 4a350ae5742b40e3926a82c692933839 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a350ae5742b40e3926a82c692933839 already in progress: creating tablet
I20251214 20:52:09.727259 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05f1ec28fb24508ae937c70aaaefffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.727774 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet e9e4c2715cae47a09f7117724ce34376 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:09.728319 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet e9e4c2715cae47a09f7117724ce34376 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9e4c2715cae47a09f7117724ce34376 already in progress: creating tablet
W20251214 20:52:09.730686 521 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac19d32a31e7405b9f16566215d6b2cd consensus state has no leader
I20251214 20:52:09.731407 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a350ae5742b40e3926a82c692933839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.732342 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ea06b0c0cb094fba91bd4b311ba90194" } modify_external_catalogs: true
I20251214 20:52:09.732568 892 tablet_service.cc:1552] Processing DeleteTablet for tablet e9e4c2715cae47a09f7117724ce34376 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.732545 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_8d8bae3c1d80444ba67b93ab0c2c406c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:09.732685 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet e9e4c2715cae47a09f7117724ce34376 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:09.733083 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d8bae3c1d80444ba67b93ab0c2c406c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.733167 1208 tablet_replica.cc:333] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:09.733393 1208 raft_consensus.cc:2243] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:09.733394 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet e9e4c2715cae47a09f7117724ce34376 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9e4c2715cae47a09f7117724ce34376 already in progress: creating tablet
I20251214 20:52:09.733599 1208 raft_consensus.cc:2272] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:09.734033 517 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6cc54be0bc6d4f259967fbe0a5023f82 consensus state has no leader
W20251214 20:52:09.734107 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 4a350ae5742b40e3926a82c692933839 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a350ae5742b40e3926a82c692933839 already in progress: creating tablet
I20251214 20:52:09.732353 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_506ad42d4328442997401fa286d1874d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.734684 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_506ad42d4328442997401fa286d1874d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.734903 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.735045 1208 ts_tablet_manager.cc:1916] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:09.735571 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 1e95f3d09dac4c2bb838d73f65df64a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e95f3d09dac4c2bb838d73f65df64a6 already in progress: creating tablet
I20251214 20:52:09.732563 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ea06b0c0cb094fba91bd4b311ba90194" } modify_external_catalogs: true
I20251214 20:52:09.736305 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:09.736907 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 23ecf83d935c49f9b304a3355b0cc55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23ecf83d935c49f9b304a3355b0cc55d already in progress: creating tablet
I20251214 20:52:09.737743 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet cefcb15849d1431fb50790343f763ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.738104 1205 tablet_replica.cc:333] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:09.738708 1205 raft_consensus.cc:2243] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:09.739231 1205 raft_consensus.cc:2272] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.740338 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e4c1451b8ea4b6f8c934633f67212b0 with delete_type TABLET_DATA_DELETED (replaced by tablet d8947ebdc87341ad8b1a01f5dd246172) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.741076 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 31f74b0b72b74085b31c16caca2efa08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.741290 1205 ts_tablet_manager.cc:1916] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:09.742257 1208 ts_tablet_manager.cc:1929] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:09.742403 1208 log.cc:1199] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/e9e4c2715cae47a09f7117724ce34376
I20251214 20:52:09.742767 1208 ts_tablet_manager.cc:1950] T e9e4c2715cae47a09f7117724ce34376 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:09.744297 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a350ae5742b40e3926a82c692933839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.744562 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 31f74b0b72b74085b31c16caca2efa08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.747109 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.747391 890 tablet_service.cc:1552] Processing DeleteTablet for tablet cefcb15849d1431fb50790343f763ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.747575 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.747839 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e4c1451b8ea4b6f8c934633f67212b0 with delete_type TABLET_DATA_DELETED (replaced by tablet d8947ebdc87341ad8b1a01f5dd246172) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.749085 521 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 47876b7d13f6455fa367933990ad2f6e (table default.Table_fcdbff382a58458bb3b9912ed28909bf [id=13be9c7a389b4137a6bcdf57d756bfa7]). Expected version 2 got 0
I20251214 20:52:09.750434 1205 ts_tablet_manager.cc:1929] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:09.750464 892 tablet_service.cc:1505] Processing CreateTablet for tablet 358d5c6b594147c8ba136b4b0732fae1 (DEFAULT_TABLE table=default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.750708 1205 log.cc:1199] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/cefcb15849d1431fb50790343f763ba1
I20251214 20:52:09.750916 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358d5c6b594147c8ba136b4b0732fae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.751263 1205 ts_tablet_manager.cc:1950] T cefcb15849d1431fb50790343f763ba1 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:09.752637 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 358d5c6b594147c8ba136b4b0732fae1 (DEFAULT_TABLE table=default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.752702 1021 tablet_service.cc:1505] Processing CreateTablet for tablet a05f1ec28fb24508ae937c70aaaefffe (DEFAULT_TABLE table=default.Table_0d1f2c5d361d4d0d9b75b90db4433e7b [id=69175aeba7894328af1e0fe73b55025d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.753093 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358d5c6b594147c8ba136b4b0732fae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.754724 517 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (table default.Table_3ddb3e05420d4c24abb69f51fece269a [id=0a848dd5f6474712b2eba70add3743dc]). Expected version 1 got 0
I20251214 20:52:09.754993 517 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:09.756170 1028 tablet_service.cc:1505] Processing CreateTablet for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 (DEFAULT_TABLE table=default.Table_eb5f20f2b40e458cade7599dbd9d5aa7 [id=e5f2cef45cbf4a69977c9be3799f9330]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.756428 890 tablet_service.cc:1505] Processing CreateTablet for tablet 5f7aff79ce154385a73c37b10b9c2430 (DEFAULT_TABLE table=default.Table_170ec365d121403eaaa13bb81ada4535 [id=4b90d1a1fbf649ce9406db32b838ed5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.756609 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f44e7223f79041ec9d66d0a1bdcd4ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.756861 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7aff79ce154385a73c37b10b9c2430. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.758173 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05f1ec28fb24508ae937c70aaaefffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.759429 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 5f7aff79ce154385a73c37b10b9c2430 (DEFAULT_TABLE table=default.Table_170ec365d121403eaaa13bb81ada4535 [id=4b90d1a1fbf649ce9406db32b838ed5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.759778 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7aff79ce154385a73c37b10b9c2430. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.759739 891 tablet_service.cc:1505] Processing CreateTablet for tablet 0c344296be0549e88d4811103eb6fee5 (DEFAULT_TABLE table=default.Table_3c3f2d9e5c2b47ed8038a7402612b093 [id=163e1e9d48bc4f51b308383354454363]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.760077 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a350ae5742b40e3926a82c692933839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.760150 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c344296be0549e88d4811103eb6fee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.760353 751 tablet_service.cc:1552] Processing DeleteTablet for tablet e9e4c2715cae47a09f7117724ce34376 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.760748 754 tablet_service.cc:1505] Processing CreateTablet for tablet 358d5c6b594147c8ba136b4b0732fae1 (DEFAULT_TABLE table=default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.761106 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358d5c6b594147c8ba136b4b0732fae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.762369 745 tablet_service.cc:1505] Processing CreateTablet for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 (DEFAULT_TABLE table=default.Table_eb5f20f2b40e458cade7599dbd9d5aa7 [id=e5f2cef45cbf4a69977c9be3799f9330]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.762775 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f44e7223f79041ec9d66d0a1bdcd4ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.763414 749 tablet_service.cc:1505] Processing CreateTablet for tablet 5f7aff79ce154385a73c37b10b9c2430 (DEFAULT_TABLE table=default.Table_170ec365d121403eaaa13bb81ada4535 [id=4b90d1a1fbf649ce9406db32b838ed5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.763432 893 tablet_service.cc:1505] Processing CreateTablet for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 (DEFAULT_TABLE table=default.Table_eb5f20f2b40e458cade7599dbd9d5aa7 [id=e5f2cef45cbf4a69977c9be3799f9330]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.763594 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05f1ec28fb24508ae937c70aaaefffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.763775 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f44e7223f79041ec9d66d0a1bdcd4ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.763787 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f7aff79ce154385a73c37b10b9c2430. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.767392 747 tablet_service.cc:1505] Processing CreateTablet for tablet 0c344296be0549e88d4811103eb6fee5 (DEFAULT_TABLE table=default.Table_3c3f2d9e5c2b47ed8038a7402612b093 [id=163e1e9d48bc4f51b308383354454363]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.767727 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c344296be0549e88d4811103eb6fee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.773542 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 0c344296be0549e88d4811103eb6fee5 (DEFAULT_TABLE table=default.Table_3c3f2d9e5c2b47ed8038a7402612b093 [id=163e1e9d48bc4f51b308383354454363]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.773908 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c344296be0549e88d4811103eb6fee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.775146 1207 ts_tablet_manager.cc:1929] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:09.775362 1207 log.cc:1199] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/6e4c1451b8ea4b6f8c934633f67212b0
I20251214 20:52:09.776038 1206 tablet_bootstrap.cc:654] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.778980 1207 ts_tablet_manager.cc:1950] T 6e4c1451b8ea4b6f8c934633f67212b0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:09.781217 1202 ts_tablet_manager.cc:1434] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.060s user 0.002s sys 0.001s
I20251214 20:52:09.781333 518 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet f44e7223f79041ec9d66d0a1bdcd4ff5
I20251214 20:52:09.784653 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 6e4c1451b8ea4b6f8c934633f67212b0 (table default.Table_544d1425256741bcb7b2496d3104d9c8 [id=a8e124ed5b7141318dfa07874e188d10]) successfully deleted
W20251214 20:52:09.784866 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 4a350ae5742b40e3926a82c692933839 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a350ae5742b40e3926a82c692933839 already in progress: creating tablet
W20251214 20:52:09.785063 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet e9e4c2715cae47a09f7117724ce34376 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9e4c2715cae47a09f7117724ce34376 already in progress: creating tablet
I20251214 20:52:09.785693 745 tablet_service.cc:1552] Processing DeleteTablet for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.785734 892 tablet_service.cc:1552] Processing DeleteTablet for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.787482 1202 tablet_bootstrap.cc:492] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:09.790190 1202 tablet_bootstrap.cc:654] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.792421 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet e9e4c2715cae47a09f7117724ce34376 (table default.Table_6cedf616f8cf45efa912b2bdf51d2fb1 [id=5f062935cc214bbd8bdcbfa43b4b48f7]) successfully deleted
I20251214 20:52:09.792879 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:09.792919 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f44e7223f79041ec9d66d0a1bdcd4ff5 already in progress: creating tablet
W20251214 20:52:09.793288 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 4a350ae5742b40e3926a82c692933839 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a350ae5742b40e3926a82c692933839 already in progress: creating tablet
W20251214 20:52:09.794472 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 31f74b0b72b74085b31c16caca2efa08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31f74b0b72b74085b31c16caca2efa08 already in progress: creating tablet
W20251214 20:52:09.795078 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 1e95f3d09dac4c2bb838d73f65df64a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e95f3d09dac4c2bb838d73f65df64a6 already in progress: creating tablet
W20251214 20:52:09.795693 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet cefcb15849d1431fb50790343f763ba1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cefcb15849d1431fb50790343f763ba1 already in progress: creating tablet
I20251214 20:52:09.795789 1206 tablet_bootstrap.cc:492] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:09.796005 1208 tablet_replica.cc:333] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:09.796010 1206 ts_tablet_manager.cc:1403] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.113s user 0.006s sys 0.008s
I20251214 20:52:09.796208 1208 raft_consensus.cc:2243] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:09.796432 1208 raft_consensus.cc:2272] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.797243 1206 raft_consensus.cc:359] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:09.797487 1206 raft_consensus.cc:385] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.797591 1206 raft_consensus.cc:740] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.798035 1206 consensus_queue.cc:260] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:09.798947 1206 ts_tablet_manager.cc:1434] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251214 20:52:09.799340 1206 tablet_bootstrap.cc:492] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:09.800220 1208 ts_tablet_manager.cc:1916] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:09.800488 1203 tablet_bootstrap.cc:492] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:09.800719 1203 ts_tablet_manager.cc:1403] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.116s user 0.010s sys 0.030s
I20251214 20:52:09.801546 1203 raft_consensus.cc:359] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.801820 1203 raft_consensus.cc:385] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.801923 1203 raft_consensus.cc:740] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.802227 1206 tablet_bootstrap.cc:654] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.802248 1203 consensus_queue.cc:260] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:09.804724 963 tablet.cc:2378] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251214 20:52:09.805459 825 tablet.cc:2378] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251214 20:52:09.807116 1203 ts_tablet_manager.cc:1434] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251214 20:52:09.808310 1203 tablet_bootstrap.cc:492] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:09.810456 1203 tablet_bootstrap.cc:654] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.809110 1208 ts_tablet_manager.cc:1929] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:09.811533 1208 log.cc:1199] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/23ecf83d935c49f9b304a3355b0cc55d
I20251214 20:52:09.811924 1208 ts_tablet_manager.cc:1950] T 23ecf83d935c49f9b304a3355b0cc55d P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:09.812829 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet cefcb15849d1431fb50790343f763ba1 (table default.Table_af25ec2aae8a46e48066cfe3f391b504 [id=5661f9fb5cc24abfa05d40d479b6fc1f]) successfully deleted
W20251214 20:52:09.813496 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 6e4c1451b8ea4b6f8c934633f67212b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e4c1451b8ea4b6f8c934633f67212b0 already in progress: creating tablet
I20251214 20:52:09.814591 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 23ecf83d935c49f9b304a3355b0cc55d (table default.Table_b934001f6c8e4e7b86b4dbcbca5d066d [id=6d4f571defdf455c8575302706c2a2b0]) successfully deleted
W20251214 20:52:09.815395 512 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 6e4c1451b8ea4b6f8c934633f67212b0 because the tablet was not found. No further retry: Not found: Tablet not found: 6e4c1451b8ea4b6f8c934633f67212b0
I20251214 20:52:09.817204 1206 tablet_bootstrap.cc:492] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:09.817394 1206 ts_tablet_manager.cc:1403] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.006s
I20251214 20:52:09.818495 1206 raft_consensus.cc:359] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.818707 1206 raft_consensus.cc:385] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.818804 1206 raft_consensus.cc:740] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.819288 1206 consensus_queue.cc:260] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:09.819749 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 31f74b0b72b74085b31c16caca2efa08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31f74b0b72b74085b31c16caca2efa08 already in progress: creating tablet
I20251214 20:52:09.820230 1206 ts_tablet_manager.cc:1434] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251214 20:52:09.820467 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f44e7223f79041ec9d66d0a1bdcd4ff5 already in progress: creating tablet
I20251214 20:52:09.820570 1206 tablet_bootstrap.cc:492] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:09.821789 1203 tablet_bootstrap.cc:492] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:09.821935 1203 ts_tablet_manager.cc:1403] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20251214 20:52:09.822341 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_8090a0016b2248f6b8fa395e4f676734" } modify_external_catalogs: true
W20251214 20:52:09.822409 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f44e7223f79041ec9d66d0a1bdcd4ff5 already in progress: creating tablet
I20251214 20:52:09.822551 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_8090a0016b2248f6b8fa395e4f676734" } modify_external_catalogs: true
I20251214 20:52:09.822708 1203 raft_consensus.cc:359] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.822897 1203 raft_consensus.cc:385] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.822994 1203 raft_consensus.cc:740] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.823295 1206 tablet_bootstrap.cc:654] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.823287 1203 consensus_queue.cc:260] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.824128 1202 tablet_bootstrap.cc:492] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:09.824368 1202 ts_tablet_manager.cc:1403] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.005s
I20251214 20:52:09.824545 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_5057f83c247b49baac131eb29130d8f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:09.824787 1203 ts_tablet_manager.cc:1434] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251214 20:52:09.825121 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5057f83c247b49baac131eb29130d8f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.825800 1203 tablet_bootstrap.cc:492] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:09.825840 1202 raft_consensus.cc:359] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.826210 1202 raft_consensus.cc:385] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.826359 1202 raft_consensus.cc:740] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.826745 1202 consensus_queue.cc:260] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.827778 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4c6523df78184e2ca1544f6a3b922ccb" } new_table_name: "default.Table_dd29a2a18b4449ee8e686081ba7f5d25" modify_external_catalogs: true
I20251214 20:52:09.828176 1203 tablet_bootstrap.cc:654] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.829203 1202 ts_tablet_manager.cc:1434] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251214 20:52:09.830327 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_9d83c341c3f8444e8b6876be854e269e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:09.830402 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_4a73de3f8186423abc9e9123ed3f1acb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.830886 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d83c341c3f8444e8b6876be854e269e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.831085 1202 tablet_bootstrap.cc:492] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:09.831089 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a73de3f8186423abc9e9123ed3f1acb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.834863 1202 tablet_bootstrap.cc:654] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.834813 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_834f1d323b2b4785a85f667e5c14e739"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:09.835005 1206 tablet_bootstrap.cc:492] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:09.835227 1206 ts_tablet_manager.cc:1403] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
W20251214 20:52:09.835513 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_834f1d323b2b4785a85f667e5c14e739 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.836143 1206 raft_consensus.cc:359] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.836418 1206 raft_consensus.cc:385] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.836499 1206 raft_consensus.cc:740] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.836808 1206 consensus_queue.cc:260] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.837396 1203 tablet_bootstrap.cc:492] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:09.837580 1203 ts_tablet_manager.cc:1403] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20251214 20:52:09.837687 1206 ts_tablet_manager.cc:1434] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:09.838081 1206 tablet_bootstrap.cc:492] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:09.838472 1203 raft_consensus.cc:359] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.838723 1203 raft_consensus.cc:385] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.838812 1203 raft_consensus.cc:740] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.839164 1203 consensus_queue.cc:260] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.840632 1206 tablet_bootstrap.cc:654] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.840945 1203 ts_tablet_manager.cc:1434] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251214 20:52:09.842345 1203 tablet_bootstrap.cc:492] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:09.844061 1203 tablet_bootstrap.cc:654] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.847313 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 69d44f933f494668b2881396e02669ab
I20251214 20:52:09.848186 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_9ceff170cc7f4bac8ae6eaf248c5e1e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.848807 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ceff170cc7f4bac8ae6eaf248c5e1e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.849772 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 69d44f933f494668b2881396e02669ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.850759 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 69d44f933f494668b2881396e02669ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.851071 1205 tablet_replica.cc:333] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:09.851302 1205 raft_consensus.cc:2243] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:09.851497 1205 raft_consensus.cc:2272] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.852293 1205 ts_tablet_manager.cc:1916] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:09.853124 1206 tablet_bootstrap.cc:492] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:09.853295 1206 ts_tablet_manager.cc:1403] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
W20251214 20:52:09.853631 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 69d44f933f494668b2881396e02669ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69d44f933f494668b2881396e02669ab already in progress: creating tablet
I20251214 20:52:09.854830 1206 raft_consensus.cc:359] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.855054 1206 raft_consensus.cc:385] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.855165 1206 raft_consensus.cc:740] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.855446 1206 consensus_queue.cc:260] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.855790 1203 tablet_bootstrap.cc:492] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:09.855754 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 69d44f933f494668b2881396e02669ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.855937 1203 ts_tablet_manager.cc:1403] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20251214 20:52:09.856062 1207 tablet_replica.cc:333] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:09.856351 1206 ts_tablet_manager.cc:1434] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:09.856667 1206 tablet_bootstrap.cc:492] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:09.856704 1203 raft_consensus.cc:359] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.856887 1203 raft_consensus.cc:385] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.856968 1203 raft_consensus.cc:740] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.856961 1207 raft_consensus.cc:2243] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:09.857156 1207 raft_consensus.cc:2272] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.857214 1203 consensus_queue.cc:260] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.858202 1207 ts_tablet_manager.cc:1916] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:09.858690 1203 ts_tablet_manager.cc:1434] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251214 20:52:09.859736 1203 tablet_bootstrap.cc:492] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:09.861567 1203 tablet_bootstrap.cc:654] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.863271 1202 tablet_bootstrap.cc:492] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:09.863547 1202 ts_tablet_manager.cc:1403] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.003s
I20251214 20:52:09.864475 1202 raft_consensus.cc:359] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.864795 1202 raft_consensus.cc:385] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.864926 1202 raft_consensus.cc:740] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.865052 1207 ts_tablet_manager.cc:1929] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:09.865191 1207 log.cc:1199] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/69d44f933f494668b2881396e02669ab
I20251214 20:52:09.865391 1202 consensus_queue.cc:260] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.865584 1207 ts_tablet_manager.cc:1950] T 69d44f933f494668b2881396e02669ab P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:09.866701 1206 tablet_bootstrap.cc:654] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.868734 1202 ts_tablet_manager.cc:1434] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251214 20:52:09.869735 1205 ts_tablet_manager.cc:1929] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:09.869932 1205 log.cc:1199] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/69d44f933f494668b2881396e02669ab
I20251214 20:52:09.870266 1202 tablet_bootstrap.cc:492] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:09.870301 1205 ts_tablet_manager.cc:1950] T 69d44f933f494668b2881396e02669ab P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:09.870959 1206 tablet_bootstrap.cc:492] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:09.871183 1206 ts_tablet_manager.cc:1403] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.005s
I20251214 20:52:09.871441 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 69d44f933f494668b2881396e02669ab (table default.Table_ea06b0c0cb094fba91bd4b311ba90194 [id=5040c9d5f4d5430dbff0ab01f2c71508]) successfully deleted
I20251214 20:52:09.872040 1206 raft_consensus.cc:359] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.872262 1206 raft_consensus.cc:385] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.872334 1206 raft_consensus.cc:740] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.872617 1206 consensus_queue.cc:260] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.874388 1206 ts_tablet_manager.cc:1434] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251214 20:52:09.875558 1206 tablet_bootstrap.cc:492] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:09.876451 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet ac19d32a31e7405b9f16566215d6b2cd
I20251214 20:52:09.877363 1206 tablet_bootstrap.cc:654] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:09.877666 520 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a0ca65d369a45958c6dac539964114b consensus state has no leader
I20251214 20:52:09.884222 892 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19d32a31e7405b9f16566215d6b2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
W20251214 20:52:09.885170 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ac19d32a31e7405b9f16566215d6b2cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac19d32a31e7405b9f16566215d6b2cd already in progress: creating tablet
I20251214 20:52:09.885463 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19d32a31e7405b9f16566215d6b2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.886323 1202 tablet_bootstrap.cc:654] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:09.887663 754 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19d32a31e7405b9f16566215d6b2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.888872 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.889273 1207 tablet_replica.cc:333] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:09.889520 1207 raft_consensus.cc:2243] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:09.889741 1207 raft_consensus.cc:2272] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.891173 749 tablet_service.cc:1552] Processing DeleteTablet for tablet e9e4c2715cae47a09f7117724ce34376 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.891429 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e4c1451b8ea4b6f8c934633f67212b0 with delete_type TABLET_DATA_DELETED (replaced by tablet d8947ebdc87341ad8b1a01f5dd246172) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.891530 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 31f74b0b72b74085b31c16caca2efa08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.891706 745 tablet_service.cc:1552] Processing DeleteTablet for tablet cefcb15849d1431fb50790343f763ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.891909 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a350ae5742b40e3926a82c692933839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.892146 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.892763 1207 ts_tablet_manager.cc:1916] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:09.895378 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ac19d32a31e7405b9f16566215d6b2cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac19d32a31e7405b9f16566215d6b2cd already in progress: creating tablet
I20251214 20:52:09.897230 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_0d1f2c5d361d4d0d9b75b90db4433e7b" } new_table_name: "default.Table_ba49861ed76e45f08158052d3e3fec58" modify_external_catalogs: true
I20251214 20:52:09.899446 521 master_service.cc:954] ReplaceTablet: received request to replace tablet cefcb15849d1431fb50790343f763ba1 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:09.899529 1207 ts_tablet_manager.cc:1929] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:09.899668 1207 log.cc:1199] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/23ecf83d935c49f9b304a3355b0cc55d
I20251214 20:52:09.900104 1207 ts_tablet_manager.cc:1950] T 23ecf83d935c49f9b304a3355b0cc55d P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:09.901643 1206 tablet_bootstrap.cc:492] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:09.901805 1206 ts_tablet_manager.cc:1403] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.001s
I20251214 20:52:09.902141 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 23ecf83d935c49f9b304a3355b0cc55d (table default.Table_b934001f6c8e4e7b86b4dbcbca5d066d [id=6d4f571defdf455c8575302706c2a2b0]) successfully deleted
I20251214 20:52:09.902642 1206 raft_consensus.cc:359] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.902839 1206 raft_consensus.cc:385] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.902936 1206 raft_consensus.cc:740] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:09.903218 1206 consensus_queue.cc:260] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.903487 1202 tablet_bootstrap.cc:492] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:09.903697 1202 ts_tablet_manager.cc:1403] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.001s
I20251214 20:52:09.904603 1207 tablet_replica.cc:333] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
W20251214 20:52:09.904661 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet e9e4c2715cae47a09f7117724ce34376 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9e4c2715cae47a09f7117724ce34376 already in progress: creating tablet
I20251214 20:52:09.904629 1202 raft_consensus.cc:359] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.904816 1207 raft_consensus.cc:2243] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:09.904955 510 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 6e4c1451b8ea4b6f8c934633f67212b0 because the tablet was not found. No further retry: Not found: Tablet not found: 6e4c1451b8ea4b6f8c934633f67212b0
I20251214 20:52:09.906222 1207 raft_consensus.cc:2272] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.904995 1202 raft_consensus.cc:385] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:09.906694 1202 raft_consensus.cc:740] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
W20251214 20:52:09.906978 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 31f74b0b72b74085b31c16caca2efa08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31f74b0b72b74085b31c16caca2efa08 already in progress: creating tablet
I20251214 20:52:09.907030 1207 ts_tablet_manager.cc:1916] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:09.907207 1202 consensus_queue.cc:260] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.910351 526 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 47876b7d13f6455fa367933990ad2f6e (table default.Table_fcdbff382a58458bb3b9912ed28909bf [id=13be9c7a389b4137a6bcdf57d756bfa7]). Expected version 2 got 0
I20251214 20:52:09.910756 1206 ts_tablet_manager.cc:1434] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20251214 20:52:09.911352 1206 tablet_bootstrap.cc:492] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:09.911571 524 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (table default.Table_3ddb3e05420d4c24abb69f51fece269a [id=0a848dd5f6474712b2eba70add3743dc]). Expected version 1 got 0
I20251214 20:52:09.915262 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet e9e4c2715cae47a09f7117724ce34376 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.915699 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 8c3b9ff3c6234507b76afd18f097b6e0 (DEFAULT_TABLE table=default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.916105 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3b9ff3c6234507b76afd18f097b6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.917023 1202 ts_tablet_manager.cc:1434] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20251214 20:52:09.918287 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 4a350ae5742b40e3926a82c692933839 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:09.918642 1202 tablet_bootstrap.cc:492] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:09.920804 892 tablet_service.cc:1505] Processing CreateTablet for tablet 8c3b9ff3c6234507b76afd18f097b6e0 (DEFAULT_TABLE table=default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:09.921036 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a05f1ec28fb24508ae937c70aaaefffe consensus state has no leader
I20251214 20:52:09.921306 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3b9ff3c6234507b76afd18f097b6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.921576 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:09.922061 756 tablet_service.cc:1505] Processing CreateTablet for tablet 8c3b9ff3c6234507b76afd18f097b6e0 (DEFAULT_TABLE table=default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.922461 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c3b9ff3c6234507b76afd18f097b6e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.923321 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.923593 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 31f74b0b72b74085b31c16caca2efa08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:09.923790 1208 tablet_replica.cc:333] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:09.923995 1208 raft_consensus.cc:2243] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:09.924180 1208 raft_consensus.cc:2272] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.924382 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 31f74b0b72b74085b31c16caca2efa08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:09.925866 895 tablet_service.cc:1505] Processing CreateTablet for tablet 8140dad1fab142d98b930b53ae5a64c2 (DEFAULT_TABLE table=default.Table_d7cbf1ffbce44772951752ecbb001da6 [id=3606c9fa9a3d4b55b86dddf293442a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.926293 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8140dad1fab142d98b930b53ae5a64c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.926473 754 tablet_service.cc:1505] Processing CreateTablet for tablet 8140dad1fab142d98b930b53ae5a64c2 (DEFAULT_TABLE table=default.Table_d7cbf1ffbce44772951752ecbb001da6 [id=3606c9fa9a3d4b55b86dddf293442a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.926873 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8140dad1fab142d98b930b53ae5a64c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.929318 890 tablet_service.cc:1505] Processing CreateTablet for tablet 47a67b1b9dc94e498cad91a72e655a75 (DEFAULT_TABLE table=default.Table_92c09424acc6404ea18b929d27d505b0 [id=16f2e0f0dbda4548871716a4a0f44dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.929740 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a67b1b9dc94e498cad91a72e655a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.929967 745 tablet_service.cc:1505] Processing CreateTablet for tablet d8947ebdc87341ad8b1a01f5dd246172 (DEFAULT_TABLE table=default.Table_544d1425256741bcb7b2496d3104d9c8 [id=a8e124ed5b7141318dfa07874e188d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.930320 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8947ebdc87341ad8b1a01f5dd246172. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.931370 1205 tablet_replica.cc:333] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:09.931850 1205 raft_consensus.cc:2243] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:09.932045 1205 raft_consensus.cc:2272] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.934669 891 tablet_service.cc:1505] Processing CreateTablet for tablet 5ba2d34a6d2645bd97fe85fed952692b (DEFAULT_TABLE table=default.Table_e23247ed295b43a1b754bf2caca42e54 [id=4b6133c53ae64267993fbf56afa38f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.935106 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba2d34a6d2645bd97fe85fed952692b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.935416 747 tablet_service.cc:1505] Processing CreateTablet for tablet 47a67b1b9dc94e498cad91a72e655a75 (DEFAULT_TABLE table=default.Table_92c09424acc6404ea18b929d27d505b0 [id=16f2e0f0dbda4548871716a4a0f44dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.935771 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a67b1b9dc94e498cad91a72e655a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.937045 1205 ts_tablet_manager.cc:1916] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:09.939540 893 tablet_service.cc:1505] Processing CreateTablet for tablet d8947ebdc87341ad8b1a01f5dd246172 (DEFAULT_TABLE table=default.Table_544d1425256741bcb7b2496d3104d9c8 [id=a8e124ed5b7141318dfa07874e188d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.939953 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8947ebdc87341ad8b1a01f5dd246172. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.930275 751 tablet_service.cc:1505] Processing CreateTablet for tablet 5ba2d34a6d2645bd97fe85fed952692b (DEFAULT_TABLE table=default.Table_e23247ed295b43a1b754bf2caca42e54 [id=4b6133c53ae64267993fbf56afa38f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.943660 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba2d34a6d2645bd97fe85fed952692b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.945643 1213 raft_consensus.cc:493] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:09.945969 1213 raft_consensus.cc:515] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:09.947638 889 tablet_service.cc:1505] Processing CreateTablet for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b (DEFAULT_TABLE table=default.Table_9ceff170cc7f4bac8ae6eaf248c5e1e9 [id=5cbe2438b4294442a9778bf1f641b60e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.948202 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.948408 1213 leader_election.cc:290] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:09.953712 749 tablet_service.cc:1505] Processing CreateTablet for tablet f2b66900d6ea42aa849c24acbbe23c40 (DEFAULT_TABLE table=default.Table_9d83c341c3f8444e8b6876be854e269e [id=64301732cd3c4699a4a9c49a0adbbfd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.954037 742 tablet_service.cc:1505] Processing CreateTablet for tablet 33a7f355b40f472883e3399209ef6762 (DEFAULT_TABLE table=default.Table_8d8bae3c1d80444ba67b93ab0c2c406c [id=5272550343144d4baaba07e97ebaa187]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.954221 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b66900d6ea42aa849c24acbbe23c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.954501 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a7f355b40f472883e3399209ef6762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.954670 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 69d44f933f494668b2881396e02669ab (table default.Table_ea06b0c0cb094fba91bd4b311ba90194 [id=5040c9d5f4d5430dbff0ab01f2c71508]) successfully deleted
W20251214 20:52:09.955014 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ac19d32a31e7405b9f16566215d6b2cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac19d32a31e7405b9f16566215d6b2cd already in progress: creating tablet
I20251214 20:52:09.957826 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_3ddb3e05420d4c24abb69f51fece269a" } new_table_name: "default.Table_efa9fa90866441c8a4d8a14d14bb4bdd" modify_external_catalogs: true
I20251214 20:52:09.957966 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_af3089686b244e669ac592b91a9e99e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:09.958130 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_e23247ed295b43a1b754bf2caca42e54" } new_table_name: "default.Table_4e95a2afe6004b6eb41370206c6faa1f" modify_external_catalogs: true
I20251214 20:52:09.957830 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_92c09424acc6404ea18b929d27d505b0" } new_table_name: "default.Table_fa0e830600ba474bbbd6a68b9593d5f2" modify_external_catalogs: true
W20251214 20:52:09.958688 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af3089686b244e669ac592b91a9e99e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.958787 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_170ec365d121403eaaa13bb81ada4535" } modify_external_catalogs: true
I20251214 20:52:09.958945 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_170ec365d121403eaaa13bb81ada4535" } modify_external_catalogs: true
I20251214 20:52:09.959256 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_506ad42d4328442997401fa286d1874d" } modify_external_catalogs: true
I20251214 20:52:09.962946 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 4a350ae5742b40e3926a82c692933839 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:09.963302 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_506ad42d4328442997401fa286d1874d" } modify_external_catalogs: true
I20251214 20:52:09.963754 751 tablet_service.cc:1505] Processing CreateTablet for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b (DEFAULT_TABLE table=default.Table_9ceff170cc7f4bac8ae6eaf248c5e1e9 [id=5cbe2438b4294442a9778bf1f641b60e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.964238 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.966676 520 master_service.cc:954] ReplaceTablet: received request to replace tablet a05f1ec28fb24508ae937c70aaaefffe from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:09.967430 745 tablet_service.cc:1505] Processing CreateTablet for tablet a8e09c13caca4a1c9018387972c9f082 (DEFAULT_TABLE table=default.Table_4a73de3f8186423abc9e9123ed3f1acb [id=397be7c374504dc5973e1cbf6b263084]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.967912 518 master_service.cc:954] ReplaceTablet: received request to replace tablet ba2ed363bbab4f1f865b183297c98daa from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:09.968897 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_6061b180a35048c4a55552330aef8b00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.969416 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6061b180a35048c4a55552330aef8b00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.977489 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e09c13caca4a1c9018387972c9f082. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.978492 1208 ts_tablet_manager.cc:1916] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:09.983392 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 8140dad1fab142d98b930b53ae5a64c2 (DEFAULT_TABLE table=default.Table_d7cbf1ffbce44772951752ecbb001da6 [id=3606c9fa9a3d4b55b86dddf293442a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.983853 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8140dad1fab142d98b930b53ae5a64c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.983987 756 tablet_service.cc:1505] Processing CreateTablet for tablet c17126f75b6241c5833269d20774c439 (DEFAULT_TABLE table=default.Table_834f1d323b2b4785a85f667e5c14e739 [id=587a162643cf4bc5a3b4c9fc0de83340]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.984427 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c17126f75b6241c5833269d20774c439. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.987326 1208 ts_tablet_manager.cc:1929] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:09.988045 1211 raft_consensus.cc:493] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:09.988306 1211 raft_consensus.cc:515] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:09.989657 522 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47a67b1b9dc94e498cad91a72e655a75 consensus state has no leader
I20251214 20:52:09.990278 742 tablet_service.cc:1505] Processing CreateTablet for tablet ebedf28e17a741169e5490822a2d95df (DEFAULT_TABLE table=default.Table_506ad42d4328442997401fa286d1874d [id=08c79b0f6de64d6397fbc34f9aa9fc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.990554 1208 log.cc:1199] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/1e95f3d09dac4c2bb838d73f65df64a6
I20251214 20:52:09.990576 1211 leader_election.cc:290] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:09.990613 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ba49861ed76e45f08158052d3e3fec58" } modify_external_catalogs: true
I20251214 20:52:09.990778 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ba49861ed76e45f08158052d3e3fec58" } modify_external_catalogs: true
I20251214 20:52:09.991000 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebedf28e17a741169e5490822a2d95df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:09.991389 1208 ts_tablet_manager.cc:1950] T 1e95f3d09dac4c2bb838d73f65df64a6 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:09.991942 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3c8ed47b57e546f59c9d863f953a17ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.992527 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c8ed47b57e546f59c9d863f953a17ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.994110 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_46ba3258cb8d46b2b0f28e6b98eb8603"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:09.994618 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46ba3258cb8d46b2b0f28e6b98eb8603 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:09.995653 747 tablet_service.cc:1505] Processing CreateTablet for tablet ecb69e385bf34709b5d954de4b660087 (DEFAULT_TABLE table=default.Table_3f3f4981717d406895af06c998f950d7 [id=49474ea35419441c9218bc8c896ffd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:09.996593 1208 tablet_replica.cc:333] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:09.997061 1208 raft_consensus.cc:2243] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:09.997350 1208 raft_consensus.cc:2272] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:09.998801 1208 ts_tablet_manager.cc:1916] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.000710 1206 tablet_bootstrap.cc:654] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.005936 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb69e385bf34709b5d954de4b660087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.000134 1202 tablet_bootstrap.cc:654] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.011487 1022 tablet_service.cc:1505] Processing CreateTablet for tablet d8947ebdc87341ad8b1a01f5dd246172 (DEFAULT_TABLE table=default.Table_544d1425256741bcb7b2496d3104d9c8 [id=a8e124ed5b7141318dfa07874e188d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.014212 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 5ba2d34a6d2645bd97fe85fed952692b (DEFAULT_TABLE table=default.Table_e23247ed295b43a1b754bf2caca42e54 [id=4b6133c53ae64267993fbf56afa38f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.016363 1025 tablet_service.cc:1505] Processing CreateTablet for tablet c17126f75b6241c5833269d20774c439 (DEFAULT_TABLE table=default.Table_834f1d323b2b4785a85f667e5c14e739 [id=587a162643cf4bc5a3b4c9fc0de83340]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.016667 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba2d34a6d2645bd97fe85fed952692b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.016803 1027 tablet_service.cc:1505] Processing CreateTablet for tablet ebedf28e17a741169e5490822a2d95df (DEFAULT_TABLE table=default.Table_506ad42d4328442997401fa286d1874d [id=08c79b0f6de64d6397fbc34f9aa9fc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.016846 1207 ts_tablet_manager.cc:1929] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.017046 1207 log.cc:1199] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/cefcb15849d1431fb50790343f763ba1
I20251214 20:52:10.017112 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebedf28e17a741169e5490822a2d95df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.017426 1207 ts_tablet_manager.cc:1950] T cefcb15849d1431fb50790343f763ba1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:10.018782 1207 tablet_replica.cc:333] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:10.018895 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet cefcb15849d1431fb50790343f763ba1 (table default.Table_af25ec2aae8a46e48066cfe3f391b504 [id=5661f9fb5cc24abfa05d40d479b6fc1f]) successfully deleted
I20251214 20:52:10.018980 1207 raft_consensus.cc:2243] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.019408 1030 tablet_service.cc:1505] Processing CreateTablet for tablet ecb69e385bf34709b5d954de4b660087 (DEFAULT_TABLE table=default.Table_3f3f4981717d406895af06c998f950d7 [id=49474ea35419441c9218bc8c896ffd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.019742 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb69e385bf34709b5d954de4b660087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.021651 1207 raft_consensus.cc:2272] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.022375 1206 tablet_bootstrap.cc:492] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.022537 1206 ts_tablet_manager.cc:1403] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.111s user 0.004s sys 0.000s
I20251214 20:52:10.022567 1207 ts_tablet_manager.cc:1916] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.023439 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8947ebdc87341ad8b1a01f5dd246172. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.023411 1206 raft_consensus.cc:359] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.023644 1206 raft_consensus.cc:385] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.023780 1206 raft_consensus.cc:740] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.024097 1206 consensus_queue.cc:260] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.025434 1206 ts_tablet_manager.cc:1434] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251214 20:52:10.025756 1206 tablet_bootstrap.cc:492] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:10.015362 1032 tablet_service.cc:1505] Processing CreateTablet for tablet a8e09c13caca4a1c9018387972c9f082 (DEFAULT_TABLE table=default.Table_4a73de3f8186423abc9e9123ed3f1acb [id=397be7c374504dc5973e1cbf6b263084]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.015727 1035 tablet_service.cc:1505] Processing CreateTablet for tablet f2b66900d6ea42aa849c24acbbe23c40 (DEFAULT_TABLE table=default.Table_9d83c341c3f8444e8b6876be854e269e [id=64301732cd3c4699a4a9c49a0adbbfd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.016037 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 33a7f355b40f472883e3399209ef6762 (DEFAULT_TABLE table=default.Table_8d8bae3c1d80444ba67b93ab0c2c406c [id=5272550343144d4baaba07e97ebaa187]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.023394 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b (DEFAULT_TABLE table=default.Table_9ceff170cc7f4bac8ae6eaf248c5e1e9 [id=5cbe2438b4294442a9778bf1f641b60e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.014887 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 47a67b1b9dc94e498cad91a72e655a75 (DEFAULT_TABLE table=default.Table_92c09424acc6404ea18b929d27d505b0 [id=16f2e0f0dbda4548871716a4a0f44dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.028534 1206 tablet_bootstrap.cc:654] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.030102 1208 ts_tablet_manager.cc:1929] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.030309 1208 log.cc:1199] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/31f74b0b72b74085b31c16caca2efa08
I20251214 20:52:10.030829 1208 ts_tablet_manager.cc:1950] T 31f74b0b72b74085b31c16caca2efa08 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:10.032907 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c17126f75b6241c5833269d20774c439. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.033284 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a67b1b9dc94e498cad91a72e655a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.033600 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a7f355b40f472883e3399209ef6762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.034956 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e09c13caca4a1c9018387972c9f082. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.037464 1206 tablet_bootstrap.cc:492] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.037690 1206 ts_tablet_manager.cc:1403] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20251214 20:52:10.038266 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.038725 1206 raft_consensus.cc:359] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.038933 1206 raft_consensus.cc:385] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.039031 1206 raft_consensus.cc:740] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.039537 1206 consensus_queue.cc:260] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.039870 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b66900d6ea42aa849c24acbbe23c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.040927 1206 ts_tablet_manager.cc:1434] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251214 20:52:10.041795 1206 tablet_bootstrap.cc:492] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:10.044348 1206 tablet_bootstrap.cc:654] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.049345 1207 ts_tablet_manager.cc:1929] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.050144 1207 log.cc:1199] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/4a350ae5742b40e3926a82c692933839
I20251214 20:52:10.050818 1207 ts_tablet_manager.cc:1950] T 4a350ae5742b40e3926a82c692933839 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:10.052678 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 4a350ae5742b40e3926a82c692933839 (table default.Table_6d8ae2f7aafa4f28a7430e353d201b63 [id=2a868a8e093247baa304d4fe1422c59a]) successfully deleted
I20251214 20:52:10.053459 1206 tablet_bootstrap.cc:492] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.053658 1206 ts_tablet_manager.cc:1403] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20251214 20:52:10.053966 1207 tablet_replica.cc:333] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:10.054387 1207 raft_consensus.cc:2243] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.054548 1206 raft_consensus.cc:359] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.055025 1207 raft_consensus.cc:2272] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.055190 1206 raft_consensus.cc:385] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.055373 1206 raft_consensus.cc:740] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.055706 1206 consensus_queue.cc:260] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.056486 747 tablet_service.cc:1505] Processing CreateTablet for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (DEFAULT_TABLE table=default.Table_5057f83c247b49baac131eb29130d8f3 [id=086ee2aba6724a93b4bb736886fe9615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.056615 1206 ts_tablet_manager.cc:1434] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251214 20:52:10.056841 1207 ts_tablet_manager.cc:1916] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.057005 1206 tablet_bootstrap.cc:492] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:10.057572 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b48bb260e854beb9ec61ce9c79b46c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.059937 1206 tablet_bootstrap.cc:654] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.065653 1205 ts_tablet_manager.cc:1929] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.065986 1205 log.cc:1199] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/e9e4c2715cae47a09f7117724ce34376
I20251214 20:52:10.066502 1205 ts_tablet_manager.cc:1950] T e9e4c2715cae47a09f7117724ce34376 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:10.066984 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (DEFAULT_TABLE table=default.Table_5057f83c247b49baac131eb29130d8f3 [id=086ee2aba6724a93b4bb736886fe9615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.067628 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b48bb260e854beb9ec61ce9c79b46c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.067762 1205 tablet_replica.cc:333] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:10.068079 1205 raft_consensus.cc:2243] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.068496 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet e9e4c2715cae47a09f7117724ce34376 (table default.Table_6cedf616f8cf45efa912b2bdf51d2fb1 [id=5f062935cc214bbd8bdcbfa43b4b48f7]) successfully deleted
I20251214 20:52:10.068964 1205 raft_consensus.cc:2272] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.069909 1205 ts_tablet_manager.cc:1916] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.076417 1207 ts_tablet_manager.cc:1929] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.076716 1207 log.cc:1199] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/1e95f3d09dac4c2bb838d73f65df64a6
I20251214 20:52:10.077142 1207 ts_tablet_manager.cc:1950] T 1e95f3d09dac4c2bb838d73f65df64a6 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:10.078498 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 1e95f3d09dac4c2bb838d73f65df64a6 (table default.Table_f46dedc9f5c44386abeb1d210d19b59e [id=53fd8efd1a1f4b7dac3c57c8e66bdc61]) successfully deleted
W20251214 20:52:10.079507 510 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 23ecf83d935c49f9b304a3355b0cc55d because the tablet was not found. No further retry: Not found: Tablet not found: 23ecf83d935c49f9b304a3355b0cc55d
I20251214 20:52:10.085836 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a0eb9ab79d9428997c6f66790d6b1a3" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:10.086438 777 raft_consensus.cc:2468] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:10.095449 891 tablet_service.cc:1505] Processing CreateTablet for tablet a8e09c13caca4a1c9018387972c9f082 (DEFAULT_TABLE table=default.Table_4a73de3f8186423abc9e9123ed3f1acb [id=397be7c374504dc5973e1cbf6b263084]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.096071 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8e09c13caca4a1c9018387972c9f082. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.111277 989 leader_election.cc:304] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:10.112082 1211 raft_consensus.cc:2804] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:10.112952 1211 raft_consensus.cc:493] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:10.113099 1211 raft_consensus.cc:3060] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.117442 1203 tablet_bootstrap.cc:492] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:10.118114 1203 ts_tablet_manager.cc:1403] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.258s user 0.035s sys 0.027s
I20251214 20:52:10.119027 1203 raft_consensus.cc:359] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:10.119797 1203 raft_consensus.cc:385] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.119899 1203 raft_consensus.cc:740] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:10.120558 524 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ba2d34a6d2645bd97fe85fed952692b consensus state has no leader
I20251214 20:52:10.120622 891 tablet_service.cc:1505] Processing CreateTablet for tablet f2b66900d6ea42aa849c24acbbe23c40 (DEFAULT_TABLE table=default.Table_9d83c341c3f8444e8b6876be854e269e [id=64301732cd3c4699a4a9c49a0adbbfd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.121176 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b66900d6ea42aa849c24acbbe23c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.122203 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_48c1e24ff9664a38a72467f0543209b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.122825 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48c1e24ff9664a38a72467f0543209b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.123397 525 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 5f7aff79ce154385a73c37b10b9c2430
I20251214 20:52:10.124049 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_fcdbff382a58458bb3b9912ed28909bf" } modify_external_catalogs: true
I20251214 20:52:10.124253 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_fcdbff382a58458bb3b9912ed28909bf" } modify_external_catalogs: true
W20251214 20:52:10.124760 523 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a0eb9ab79d9428997c6f66790d6b1a3 consensus state has no leader
I20251214 20:52:10.125083 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4a73de3f8186423abc9e9123ed3f1acb" } modify_external_catalogs: true
I20251214 20:52:10.125214 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4a73de3f8186423abc9e9123ed3f1acb" } modify_external_catalogs: true
I20251214 20:52:10.126442 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet ebedf28e17a741169e5490822a2d95df
I20251214 20:52:10.127102 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_850408f19d2848309b88410e331f0798"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.127846 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_850408f19d2848309b88410e331f0798 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.129192 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet a05f1ec28fb24508ae937c70aaaefffe
I20251214 20:52:10.129593 520 catalog_manager.cc:6724] ReplaceTablet: tablet a05f1ec28fb24508ae937c70aaaefffe deleted and replaced by tablet 8e2ee5cc95304fc1a80ed4d31838ac38
I20251214 20:52:10.129923 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_9d83c341c3f8444e8b6876be854e269e" } new_table_name: "default.Table_5f7097b36879493d8b5fdfcf1d752ce3" modify_external_catalogs: true
I20251214 20:52:10.130703 892 tablet_service.cc:1505] Processing CreateTablet for tablet 33a7f355b40f472883e3399209ef6762 (DEFAULT_TABLE table=default.Table_8d8bae3c1d80444ba67b93ab0c2c406c [id=5272550343144d4baaba07e97ebaa187]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.131233 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a7f355b40f472883e3399209ef6762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.130602 1203 consensus_queue.cc:260] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:10.131392 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_89dffab0bf1d4856b671651a07b3439e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.131853 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89dffab0bf1d4856b671651a07b3439e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.132718 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47876b7d13f6455fa367933990ad2f6e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:10.134275 1205 ts_tablet_manager.cc:1929] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251214 20:52:10.134888 851 leader_election.cc:343] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.139279 892 tablet_service.cc:1505] Processing CreateTablet for tablet c17126f75b6241c5833269d20774c439 (DEFAULT_TABLE table=default.Table_834f1d323b2b4785a85f667e5c14e739 [id=587a162643cf4bc5a3b4c9fc0de83340]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.139765 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c17126f75b6241c5833269d20774c439. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.141167 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7aff79ce154385a73c37b10b9c2430 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.141310 1205 log.cc:1199] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/31f74b0b72b74085b31c16caca2efa08
I20251214 20:52:10.141461 756 tablet_service.cc:1552] Processing DeleteTablet for tablet ebedf28e17a741169e5490822a2d95df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.141673 742 tablet_service.cc:1552] Processing DeleteTablet for tablet a05f1ec28fb24508ae937c70aaaefffe with delete_type TABLET_DATA_DELETED (replaced by tablet 8e2ee5cc95304fc1a80ed4d31838ac38) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.142278 1205 ts_tablet_manager.cc:1950] T 31f74b0b72b74085b31c16caca2efa08 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:10.142480 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 5f7aff79ce154385a73c37b10b9c2430 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f7aff79ce154385a73c37b10b9c2430 already in progress: creating tablet
W20251214 20:52:10.143153 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ebedf28e17a741169e5490822a2d95df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebedf28e17a741169e5490822a2d95df already in progress: creating tablet
I20251214 20:52:10.142513 1211 raft_consensus.cc:515] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:10.143721 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet a05f1ec28fb24508ae937c70aaaefffe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a05f1ec28fb24508ae937c70aaaefffe already in progress: creating tablet
I20251214 20:52:10.143921 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet ebedf28e17a741169e5490822a2d95df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.144121 1211 leader_election.cc:290] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:10.144219 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet a05f1ec28fb24508ae937c70aaaefffe with delete_type TABLET_DATA_DELETED (replaced by tablet 8e2ee5cc95304fc1a80ed4d31838ac38) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.144556 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a0eb9ab79d9428997c6f66790d6b1a3" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:10.144716 518 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet ba2ed363bbab4f1f865b183297c98daa
I20251214 20:52:10.144788 777 raft_consensus.cc:3060] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.145249 518 catalog_manager.cc:6724] ReplaceTablet: tablet ba2ed363bbab4f1f865b183297c98daa deleted and replaced by tablet 569a02142c71427d91868b02695ae12b
I20251214 20:52:10.147245 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7aff79ce154385a73c37b10b9c2430 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.147562 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet ba2ed363bbab4f1f865b183297c98daa with delete_type TABLET_DATA_DELETED (replaced by tablet 569a02142c71427d91868b02695ae12b) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.147887 742 tablet_service.cc:1552] Processing DeleteTablet for tablet ba2ed363bbab4f1f865b183297c98daa with delete_type TABLET_DATA_DELETED (replaced by tablet 569a02142c71427d91868b02695ae12b) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.148824 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 31f74b0b72b74085b31c16caca2efa08 (table default.Table_bd2058c3daa94a0cbabdda39dbf5ec2b [id=fe9038b8f14d46e8aadc6bf34505b684]) successfully deleted
I20251214 20:52:10.148154 777 raft_consensus.cc:2468] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:10.149820 989 leader_election.cc:304] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
W20251214 20:52:10.149883 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ebedf28e17a741169e5490822a2d95df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebedf28e17a741169e5490822a2d95df already in progress: creating tablet
I20251214 20:52:10.150264 1211 raft_consensus.cc:2804] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:10.150473 1211 raft_consensus.cc:697] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
W20251214 20:52:10.150494 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a05f1ec28fb24508ae937c70aaaefffe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a05f1ec28fb24508ae937c70aaaefffe already in progress: creating tablet
W20251214 20:52:10.151011 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ba2ed363bbab4f1f865b183297c98daa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba2ed363bbab4f1f865b183297c98daa already in progress: creating tablet
W20251214 20:52:10.151317 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ba2ed363bbab4f1f865b183297c98daa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba2ed363bbab4f1f865b183297c98daa already in progress: creating tablet
I20251214 20:52:10.151460 1211 consensus_queue.cc:237] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:10.151868 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 5f7aff79ce154385a73c37b10b9c2430 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f7aff79ce154385a73c37b10b9c2430 already in progress: creating tablet
I20251214 20:52:10.160058 895 tablet_service.cc:1505] Processing CreateTablet for tablet ebedf28e17a741169e5490822a2d95df (DEFAULT_TABLE table=default.Table_506ad42d4328442997401fa286d1874d [id=08c79b0f6de64d6397fbc34f9aa9fc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.160656 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebedf28e17a741169e5490822a2d95df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.169310 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7aff79ce154385a73c37b10b9c2430 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.170162 890 tablet_service.cc:1552] Processing DeleteTablet for tablet ebedf28e17a741169e5490822a2d95df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.172415 890 tablet_service.cc:1552] Processing DeleteTablet for tablet a05f1ec28fb24508ae937c70aaaefffe with delete_type TABLET_DATA_DELETED (replaced by tablet 8e2ee5cc95304fc1a80ed4d31838ac38) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.173075 890 tablet_service.cc:1552] Processing DeleteTablet for tablet ba2ed363bbab4f1f865b183297c98daa with delete_type TABLET_DATA_DELETED (replaced by tablet 569a02142c71427d91868b02695ae12b) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.175014 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 1e95f3d09dac4c2bb838d73f65df64a6 (table default.Table_f46dedc9f5c44386abeb1d210d19b59e [id=53fd8efd1a1f4b7dac3c57c8e66bdc61]) successfully deleted
I20251214 20:52:10.175335 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 31f74b0b72b74085b31c16caca2efa08 (table default.Table_bd2058c3daa94a0cbabdda39dbf5ec2b [id=fe9038b8f14d46e8aadc6bf34505b684]) successfully deleted
W20251214 20:52:10.175958 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 5f7aff79ce154385a73c37b10b9c2430 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f7aff79ce154385a73c37b10b9c2430 already in progress: creating tablet
W20251214 20:52:10.176191 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ebedf28e17a741169e5490822a2d95df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebedf28e17a741169e5490822a2d95df already in progress: creating tablet
W20251214 20:52:10.176352 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a05f1ec28fb24508ae937c70aaaefffe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a05f1ec28fb24508ae937c70aaaefffe already in progress: creating tablet
W20251214 20:52:10.176553 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ba2ed363bbab4f1f865b183297c98daa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba2ed363bbab4f1f865b183297c98daa already in progress: creating tablet
I20251214 20:52:10.177006 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47876b7d13f6455fa367933990ad2f6e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:10.177978 895 tablet_service.cc:1505] Processing CreateTablet for tablet ecb69e385bf34709b5d954de4b660087 (DEFAULT_TABLE table=default.Table_3f3f4981717d406895af06c998f950d7 [id=49474ea35419441c9218bc8c896ffd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.179670 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb69e385bf34709b5d954de4b660087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.183200 1206 tablet_bootstrap.cc:492] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.183462 1206 ts_tablet_manager.cc:1403] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.127s user 0.018s sys 0.051s
I20251214 20:52:10.184459 1206 raft_consensus.cc:359] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.184747 1206 raft_consensus.cc:385] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.185050 1206 raft_consensus.cc:740] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.185441 1206 consensus_queue.cc:260] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.187471 1206 ts_tablet_manager.cc:1434] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251214 20:52:10.187790 1206 tablet_bootstrap.cc:492] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:10.188894 1202 tablet_bootstrap.cc:492] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:10.189078 1202 ts_tablet_manager.cc:1403] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.271s user 0.023s sys 0.027s
I20251214 20:52:10.189762 1202 raft_consensus.cc:359] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:10.189945 1202 raft_consensus.cc:385] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.190078 1202 raft_consensus.cc:740] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.190301 1202 consensus_queue.cc:260] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:10.190938 1202 ts_tablet_manager.cc:1434] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:10.191387 1202 tablet_bootstrap.cc:492] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:10.191620 848 leader_election.cc:343] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.191838 848 leader_election.cc:304] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:10.192430 1213 raft_consensus.cc:2749] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:10.194952 1202 tablet_bootstrap.cc:654] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.200691 1217 raft_consensus.cc:493] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:10.200943 1217 raft_consensus.cc:515] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.203351 1217 leader_election.cc:290] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:10.218187 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47876b7d13f6455fa367933990ad2f6e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:10.221249 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_a5388ce81c77421aa5c30bfe834f60d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.221818 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5388ce81c77421aa5c30bfe834f60d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.222483 1206 tablet_bootstrap.cc:654] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.224565 1202 tablet_bootstrap.cc:492] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:10.225419 1202 ts_tablet_manager.cc:1403] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.003s
I20251214 20:52:10.226548 1206 tablet_bootstrap.cc:492] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.227155 1206 ts_tablet_manager.cc:1403] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.002s
I20251214 20:52:10.228201 1206 raft_consensus.cc:359] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.228425 1206 raft_consensus.cc:385] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.228502 1206 raft_consensus.cc:740] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.228901 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_e8acf2f630434ce3b7f06a4a01e4a2a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:10.229295 892 tablet_service.cc:1505] Processing CreateTablet for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (DEFAULT_TABLE table=default.Table_5057f83c247b49baac131eb29130d8f3 [id=086ee2aba6724a93b4bb736886fe9615]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:10.229472 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8acf2f630434ce3b7f06a4a01e4a2a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.229621 1202 raft_consensus.cc:359] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.229836 1202 raft_consensus.cc:385] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.229933 1202 raft_consensus.cc:740] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.230162 1202 consensus_queue.cc:260] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.230262 523 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet a8e09c13caca4a1c9018387972c9f082
I20251214 20:52:10.230249 1206 consensus_queue.cc:260] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.231024 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_0051db617c204791a9ed8d4795676c79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.231680 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0051db617c204791a9ed8d4795676c79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.231688 1202 ts_tablet_manager.cc:1434] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20251214 20:52:10.226940 713 leader_election.cc:343] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.233014 1206 ts_tablet_manager.cc:1434] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251214 20:52:10.233201 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a0eb9ab79d9428997c6f66790d6b1a3 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.233357 1206 tablet_bootstrap.cc:492] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:10.233681 520 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2b66900d6ea42aa849c24acbbe23c40 consensus state has no leader
I20251214 20:52:10.233935 520 master_service.cc:954] ReplaceTablet: received request to replace tablet d8947ebdc87341ad8b1a01f5dd246172 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.234336 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a350ae5742b40e3926a82c692933839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.234606 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet a8e09c13caca4a1c9018387972c9f082 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.235457 1206 tablet_bootstrap.cc:654] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.235386 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_8d8bae3c1d80444ba67b93ab0c2c406c" } new_table_name: "default.Table_1c75ca10deb842669e2fea0f16ecc983" modify_external_catalogs: true
W20251214 20:52:10.235699 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 4a350ae5742b40e3926a82c692933839 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a350ae5742b40e3926a82c692933839 already in progress: creating tablet
W20251214 20:52:10.235996 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a8e09c13caca4a1c9018387972c9f082 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e09c13caca4a1c9018387972c9f082 already in progress: creating tablet
I20251214 20:52:10.239379 890 tablet_service.cc:1552] Processing DeleteTablet for tablet a8e09c13caca4a1c9018387972c9f082 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.240033 1203 ts_tablet_manager.cc:1434] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.122s user 0.029s sys 0.020s
I20251214 20:52:10.240455 1203 tablet_bootstrap.cc:492] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:10.242385 1203 tablet_bootstrap.cc:654] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.243527 742 tablet_service.cc:1552] Processing DeleteTablet for tablet a8e09c13caca4a1c9018387972c9f082 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
W20251214 20:52:10.244315 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet a8e09c13caca4a1c9018387972c9f082 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e09c13caca4a1c9018387972c9f082 already in progress: creating tablet
I20251214 20:52:10.249794 1203 tablet_bootstrap.cc:492] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:10.249966 1203 ts_tablet_manager.cc:1403] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20251214 20:52:10.250851 1203 raft_consensus.cc:359] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.251051 1203 raft_consensus.cc:385] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.251159 1203 raft_consensus.cc:740] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.251437 1203 consensus_queue.cc:260] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.251976 1203 ts_tablet_manager.cc:1434] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:10.252305 1203 tablet_bootstrap.cc:492] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:10.254184 1203 tablet_bootstrap.cc:654] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.257834 1203 tablet_bootstrap.cc:492] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:10.258072 1203 ts_tablet_manager.cc:1403] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20251214 20:52:10.258898 1203 raft_consensus.cc:359] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.259092 1203 raft_consensus.cc:385] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.259207 1203 raft_consensus.cc:740] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.259441 1203 consensus_queue.cc:260] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.260056 1203 ts_tablet_manager.cc:1434] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:10.260390 1203 tablet_bootstrap.cc:492] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:10.261065 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 8c3b9ff3c6234507b76afd18f097b6e0 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.261731 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_544d1425256741bcb7b2496d3104d9c8" } new_table_name: "default.Table_b971804db7ad428a88f22d83a1de29d9" modify_external_catalogs: true
I20251214 20:52:10.263043 1202 tablet_bootstrap.cc:492] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:10.264338 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a8e09c13caca4a1c9018387972c9f082 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8e09c13caca4a1c9018387972c9f082 already in progress: creating tablet
I20251214 20:52:10.264878 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet d8947ebdc87341ad8b1a01f5dd246172
I20251214 20:52:10.265137 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_e3603e6d524c40dea2a1a88b79d63a8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:10.265487 520 catalog_manager.cc:6724] ReplaceTablet: tablet d8947ebdc87341ad8b1a01f5dd246172 deleted and replaced by tablet d13f93bb5e5f4b59bcf9f4bfed070f25
W20251214 20:52:10.265807 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3603e6d524c40dea2a1a88b79d63a8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.266007 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet d8947ebdc87341ad8b1a01f5dd246172 with delete_type TABLET_DATA_DELETED (replaced by tablet d13f93bb5e5f4b59bcf9f4bfed070f25) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:10.266660 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet d8947ebdc87341ad8b1a01f5dd246172 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8947ebdc87341ad8b1a01f5dd246172 already in progress: creating tablet
I20251214 20:52:10.267338 891 tablet_service.cc:1552] Processing DeleteTablet for tablet d8947ebdc87341ad8b1a01f5dd246172 with delete_type TABLET_DATA_DELETED (replaced by tablet d13f93bb5e5f4b59bcf9f4bfed070f25) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.267560 742 tablet_service.cc:1552] Processing DeleteTablet for tablet d8947ebdc87341ad8b1a01f5dd246172 with delete_type TABLET_DATA_DELETED (replaced by tablet d13f93bb5e5f4b59bcf9f4bfed070f25) from {username='slave'} at 127.0.0.1:37374
W20251214 20:52:10.267624 1181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251214 20:52:10.268034 1181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 20:52:10.268136 1181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251214 20:52:10.268316 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet d8947ebdc87341ad8b1a01f5dd246172 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8947ebdc87341ad8b1a01f5dd246172 already in progress: creating tablet
W20251214 20:52:10.268457 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet d8947ebdc87341ad8b1a01f5dd246172 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8947ebdc87341ad8b1a01f5dd246172 already in progress: creating tablet
I20251214 20:52:10.270560 1203 tablet_bootstrap.cc:654] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.270761 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a350ae5742b40e3926a82c692933839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.271065 1208 tablet_replica.cc:333] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:10.271338 1208 raft_consensus.cc:2243] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.271564 1208 raft_consensus.cc:2272] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.272969 1208 ts_tablet_manager.cc:1916] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:10.275942 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f2b66900d6ea42aa849c24acbbe23c40 (table default.Table_5f7097b36879493d8b5fdfcf1d752ce3 [id=64301732cd3c4699a4a9c49a0adbbfd4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f2b66900d6ea42aa849c24acbbe23c40 consensus state has no leader
W20251214 20:52:10.277838 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33a7f355b40f472883e3399209ef6762 consensus state has no leader
W20251214 20:52:10.277895 521 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d13f93bb5e5f4b59bcf9f4bfed070f25 has no consensus state
W20251214 20:52:10.280555 1181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 20:52:10.280773 1181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 20:52:10.280908 1181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 20:52:10.281030 1181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 20:52:10.283716 1250 raft_consensus.cc:493] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:10.283885 1250 raft_consensus.cc:515] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.284796 1250 leader_election.cc:290] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:10.285426 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bfb575249b8403991d21c6fc96eefeb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:10.285753 777 raft_consensus.cc:2468] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:10.286410 989 leader_election.cc:304] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:10.287155 1250 raft_consensus.cc:2804] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:10.287276 1250 raft_consensus.cc:493] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:10.287349 1250 raft_consensus.cc:3060] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.289194 895 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd0fa0dad974c98bf6f52c8cfe83035 (DEFAULT_TABLE table=default.Table_af3089686b244e669ac592b91a9e99e6 [id=5c14810dd5364dcaa218b80c3756eadc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.289376 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd0fa0dad974c98bf6f52c8cfe83035 (DEFAULT_TABLE table=default.Table_af3089686b244e669ac592b91a9e99e6 [id=5c14810dd5364dcaa218b80c3756eadc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.289798 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd0fa0dad974c98bf6f52c8cfe83035. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.290280 742 tablet_service.cc:1505] Processing CreateTablet for tablet 3fd0fa0dad974c98bf6f52c8cfe83035 (DEFAULT_TABLE table=default.Table_af3089686b244e669ac592b91a9e99e6 [id=5c14810dd5364dcaa218b80c3756eadc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.290719 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd0fa0dad974c98bf6f52c8cfe83035. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.297847 1250 raft_consensus.cc:515] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.298899 1250 leader_election.cc:290] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:10.299141 1181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=40473
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 14 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9516
ASAN enabled
I20251214 20:52:10.300668 517 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 8e2ee5cc95304fc1a80ed4d31838ac38
I20251214 20:52:10.301694 1181 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 20:52:10.304566 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bfb575249b8403991d21c6fc96eefeb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:10.304826 1181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251214 20:52:10.304841 777 raft_consensus.cc:3060] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.308143 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bfb575249b8403991d21c6fc96eefeb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:10.308564 912 raft_consensus.cc:3060] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.308921 777 raft_consensus.cc:2468] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:10.309646 989 leader_election.cc:304] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:10.310012 1250 raft_consensus.cc:2804] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:10.310153 1250 raft_consensus.cc:697] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:10.310360 1203 tablet_bootstrap.cc:492] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:10.310528 1203 ts_tablet_manager.cc:1403] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.001s
I20251214 20:52:10.310477 1250 consensus_queue.cc:237] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.311385 1203 raft_consensus.cc:359] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.311638 1203 raft_consensus.cc:385] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.311728 1203 raft_consensus.cc:740] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.312017 1203 consensus_queue.cc:260] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.312638 1203 ts_tablet_manager.cc:1434] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251214 20:52:10.312932 1203 tablet_bootstrap.cc:492] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:10.314765 1203 tablet_bootstrap.cc:654] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.318359 912 raft_consensus.cc:2468] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:10.321162 524 catalog_manager.cc:5654] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:10.322392 524 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (table default.Table_efa9fa90866441c8a4d8a14d14bb4bdd [id=0a848dd5f6474712b2eba70add3743dc]). Expected version 2 got 0
I20251214 20:52:10.324354 756 tablet_service.cc:1552] Processing DeleteTablet for tablet e9e4c2715cae47a09f7117724ce34376 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.324728 1207 tablet_replica.cc:333] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:10.324931 1207 raft_consensus.cc:2243] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.325142 1207 raft_consensus.cc:2272] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.325997 1207 ts_tablet_manager.cc:1916] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.326678 521 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:10.330408 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 8c3b9ff3c6234507b76afd18f097b6e0
W20251214 20:52:10.330399 1267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:10.330935 522 catalog_manager.cc:6724] ReplaceTablet: tablet 8c3b9ff3c6234507b76afd18f097b6e0 deleted and replaced by tablet 649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:10.332158 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c3b9ff3c6234507b76afd18f097b6e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 649f5bd67cbb4f61b7066269f0bf9266) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.333072 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c3b9ff3c6234507b76afd18f097b6e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 649f5bd67cbb4f61b7066269f0bf9266) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.333271 1250 raft_consensus.cc:493] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:10.333460 1250 raft_consensus.cc:515] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
W20251214 20:52:10.333577 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8c3b9ff3c6234507b76afd18f097b6e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c3b9ff3c6234507b76afd18f097b6e0 already in progress: creating tablet
I20251214 20:52:10.334259 1250 leader_election.cc:290] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:10.334784 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c3b9ff3c6234507b76afd18f097b6e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 649f5bd67cbb4f61b7066269f0bf9266) from {username='slave'} at 127.0.0.1:37374
W20251214 20:52:10.335188 1268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:10.337350 525 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 47876b7d13f6455fa367933990ad2f6e
I20251214 20:52:10.337591 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a350ae5742b40e3926a82c692933839" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:10.340785 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 47876b7d13f6455fa367933990ad2f6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.340932 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_452ec7356ff74de288f4a0fdfbc09dca" } modify_external_catalogs: true
I20251214 20:52:10.341179 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_452ec7356ff74de288f4a0fdfbc09dca" } modify_external_catalogs: true
I20251214 20:52:10.341647 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 31f74b0b72b74085b31c16caca2efa08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.341945 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 47876b7d13f6455fa367933990ad2f6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.339550 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 47876b7d13f6455fa367933990ad2f6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:10.338495 991 leader_election.cc:343] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: Tablet not RUNNING: STOPPED
W20251214 20:52:10.343390 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 47876b7d13f6455fa367933990ad2f6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47876b7d13f6455fa367933990ad2f6e already in progress: creating tablet
I20251214 20:52:10.343883 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_5f7097b36879493d8b5fdfcf1d752ce3" } new_table_name: "default.Table_934b5fc4d4004b68b7a77ec2cfd47f8c" modify_external_catalogs: true
I20251214 20:52:10.343992 1207 ts_tablet_manager.cc:1929] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.344220 1207 log.cc:1199] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/e9e4c2715cae47a09f7117724ce34376
I20251214 20:52:10.340790 1203 tablet_bootstrap.cc:492] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:10.345093 1203 ts_tablet_manager.cc:1403] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.007s
I20251214 20:52:10.345993 1203 raft_consensus.cc:359] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.346117 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_850408f19d2848309b88410e331f0798" } modify_external_catalogs: true
I20251214 20:52:10.346222 1203 raft_consensus.cc:385] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.346266 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_850408f19d2848309b88410e331f0798" } modify_external_catalogs: true
I20251214 20:52:10.346292 1203 raft_consensus.cc:740] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.346524 1203 consensus_queue.cc:260] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.347003 1203 ts_tablet_manager.cc:1434] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:10.347810 523 master_service.cc:954] ReplaceTablet: received request to replace tablet a8e09c13caca4a1c9018387972c9f082 from {username='slave'} at 127.0.0.1:48876
W20251214 20:52:10.347960 989 leader_election.cc:343] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 4a350ae5742b40e3926a82c692933839
I20251214 20:52:10.348106 989 leader_election.cc:304] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:10.228932 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a0eb9ab79d9428997c6f66790d6b1a3" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:10.229168 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a0eb9ab79d9428997c6f66790d6b1a3" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:10.349792 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_348a96aca6644d08b3074727ff3df5a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.350381 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_348a96aca6644d08b3074727ff3df5a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.350515 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b48bb260e854beb9ec61ce9c79b46c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.351040 1250 raft_consensus.cc:2749] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:10.351785 1203 tablet_bootstrap.cc:492] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:10.353766 1203 tablet_bootstrap.cc:654] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.287600 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bfb575249b8403991d21c6fc96eefeb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:10.355564 913 raft_consensus.cc:2376] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1. Re-sending same reply.
I20251214 20:52:10.356446 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_3f3f4981717d406895af06c998f950d7" } new_table_name: "default.Table_500b29027c11444893ae5b7a3123cb8b" modify_external_catalogs: true
I20251214 20:52:10.358021 1181 server_base.cc:1047] running on GCE node
I20251214 20:52:10.359340 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd0fa0dad974c98bf6f52c8cfe83035. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.359607 523 master_service.cc:954] ReplaceTablet: received request to replace tablet f44e7223f79041ec9d66d0a1bdcd4ff5 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.359625 1217 raft_consensus.cc:493] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:10.359792 1217 raft_consensus.cc:515] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.360457 1217 leader_election.cc:290] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:10.361122 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6381cbf05a8c4f84b71f229497fd28ef" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:10.361357 1056 raft_consensus.cc:2468] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:10.361917 713 leader_election.cc:304] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:10.362430 1217 raft_consensus.cc:2804] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:10.362520 1217 raft_consensus.cc:493] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:10.362622 1217 raft_consensus.cc:3060] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.365051 1202 tablet_bootstrap.cc:654] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.365566 1217 raft_consensus.cc:515] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.367149 1208 ts_tablet_manager.cc:1929] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.367509 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 35101a6997e24040994661756d9649d4
W20251214 20:52:10.367827 991 leader_election.cc:343] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:10.368225 991 leader_election.cc:343] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.369475 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 35101a6997e24040994661756d9649d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:10.369613 525 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2b66900d6ea42aa849c24acbbe23c40 consensus state has no leader
W20251214 20:52:10.370110 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 35101a6997e24040994661756d9649d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35101a6997e24040994661756d9649d4 already in progress: creating tablet
I20251214 20:52:10.370168 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 35101a6997e24040994661756d9649d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.370678 518 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 810ba69dda7f4631b1432bd99029609b
I20251214 20:52:10.372180 1207 ts_tablet_manager.cc:1950] T e9e4c2715cae47a09f7117724ce34376 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:10.372946 1217 leader_election.cc:290] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:10.373174 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6381cbf05a8c4f84b71f229497fd28ef" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:10.373416 1056 raft_consensus.cc:3060] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.373705 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 810ba69dda7f4631b1432bd99029609b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:10.374611 513 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 810ba69dda7f4631b1432bd99029609b because the tablet was not found. No further retry: Not found: Tablet not found: 810ba69dda7f4631b1432bd99029609b
I20251214 20:52:10.375101 1203 tablet_bootstrap.cc:492] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:10.374984 518 catalog_manager.cc:5654] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:10.375310 1203 ts_tablet_manager.cc:1403] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.006s
I20251214 20:52:10.376629 1203 raft_consensus.cc:359] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.376845 1203 raft_consensus.cc:385] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.376960 1203 raft_consensus.cc:740] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.377166 1056 raft_consensus.cc:2468] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:10.377542 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 810ba69dda7f4631b1432bd99029609b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.378340 713 leader_election.cc:304] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:10.378396 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 2a0eb9ab79d9428997c6f66790d6b1a3
I20251214 20:52:10.378683 1217 raft_consensus.cc:2804] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:10.377228 1203 consensus_queue.cc:260] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.378903 519 catalog_manager.cc:6724] ReplaceTablet: tablet 2a0eb9ab79d9428997c6f66790d6b1a3 deleted and replaced by tablet a142157acbcd44adb6099b4428103ee5
I20251214 20:52:10.379918 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a142157acbcd44adb6099b4428103ee5) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.380229 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a142157acbcd44adb6099b4428103ee5) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.380532 1205 tablet_replica.cc:333] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:10.380749 521 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ecb69e385bf34709b5d954de4b660087 consensus state has no leader
I20251214 20:52:10.380744 1205 raft_consensus.cc:2243] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:10.381289 1205 pending_rounds.cc:70] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:10.381569 1205 pending_rounds.cc:77] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493691507585024 op_type: NO_OP noop_request { }
I20251214 20:52:10.381827 1205 raft_consensus.cc:2889] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:10.381958 1205 raft_consensus.cc:2272] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.383193 1203 ts_tablet_manager.cc:1434] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20251214 20:52:10.383636 1271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:10.384194 1203 tablet_bootstrap.cc:492] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:10.385056 1181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 20:52:10.386870 1208 log.cc:1199] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/4a350ae5742b40e3926a82c692933839
I20251214 20:52:10.387481 1211 raft_consensus.cc:493] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:10.391515 1211 raft_consensus.cc:515] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.391680 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet e9e4c2715cae47a09f7117724ce34376 (table default.Table_6cedf616f8cf45efa912b2bdf51d2fb1 [id=5f062935cc214bbd8bdcbfa43b4b48f7]) successfully deleted
I20251214 20:52:10.391778 680 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 810ba69dda7f4631b1432bd99029609b
I20251214 20:52:10.392269 1211 leader_election.cc:290] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:10.392622 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8c3b9ff3c6234507b76afd18f097b6e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c3b9ff3c6234507b76afd18f097b6e0 already in progress: creating tablet
I20251214 20:52:10.382815 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 35101a6997e24040994661756d9649d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.392999 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19d32a31e7405b9f16566215d6b2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.393240 1207 tablet_replica.cc:333] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:10.393465 1207 raft_consensus.cc:2243] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.393652 1207 raft_consensus.cc:2272] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.393652 1205 ts_tablet_manager.cc:1916] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.394183 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e95f3d09dac4c2bb838d73f65df64a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.395269 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 47876b7d13f6455fa367933990ad2f6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.395632 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a350ae5742b40e3926a82c692933839 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.395887 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 810ba69dda7f4631b1432bd99029609b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.398500 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e4c1451b8ea4b6f8c934633f67212b0 with delete_type TABLET_DATA_DELETED (replaced by tablet d8947ebdc87341ad8b1a01f5dd246172) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.399463 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44f51e11041841e98b9a15d380226e4e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:10.399763 777 raft_consensus.cc:2468] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:10.399994 1207 ts_tablet_manager.cc:1916] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.400542 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 810ba69dda7f4631b1432bd99029609b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.400924 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 810ba69dda7f4631b1432bd99029609b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.401364 989 leader_election.cc:304] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:10.401705 1211 raft_consensus.cc:2804] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:10.401834 1211 raft_consensus.cc:493] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:10.401937 1211 raft_consensus.cc:3060] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.401930 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 810ba69dda7f4631b1432bd99029609b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.402311 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44f51e11041841e98b9a15d380226e4e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:10.403374 891 tablet_service.cc:1505] Processing CreateTablet for tablet ed1d007f1ca24b7291262708ddfca3c3 (DEFAULT_TABLE table=default.Table_e3603e6d524c40dea2a1a88b79d63a8e [id=7febb36ade704d57bdfd4211054277ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.403909 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed1d007f1ca24b7291262708ddfca3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.404758 1211 raft_consensus.cc:515] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.406394 1181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 20:52:10.406430 1211 leader_election.cc:290] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:10.406486 1031 tablet_service.cc:1505] Processing CreateTablet for tablet ed1d007f1ca24b7291262708ddfca3c3 (DEFAULT_TABLE table=default.Table_e3603e6d524c40dea2a1a88b79d63a8e [id=7febb36ade704d57bdfd4211054277ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.406769 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44f51e11041841e98b9a15d380226e4e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:10.406913 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed1d007f1ca24b7291262708ddfca3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.407115 777 raft_consensus.cc:3060] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.407974 1181 hybrid_clock.cc:648] HybridClock initialized: now 1765745530407897 us; error 77 us; skew 500 ppm
I20251214 20:52:10.408497 1181 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 20:52:10.409968 777 raft_consensus.cc:2468] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:10.410379 1208 ts_tablet_manager.cc:1950] T 4a350ae5742b40e3926a82c692933839 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:10.410544 989 leader_election.cc:304] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:10.410977 1211 raft_consensus.cc:2804] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:10.411531 1211 raft_consensus.cc:697] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:10.411769 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 4a350ae5742b40e3926a82c692933839 (table default.Table_6d8ae2f7aafa4f28a7430e353d201b63 [id=2a868a8e093247baa304d4fe1422c59a]) successfully deleted
I20251214 20:52:10.411926 1208 tablet_replica.cc:333] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:10.411933 1211 consensus_queue.cc:237] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.412142 1208 raft_consensus.cc:2243] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:10.412184 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8c3b9ff3c6234507b76afd18f097b6e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c3b9ff3c6234507b76afd18f097b6e0 already in progress: creating tablet
I20251214 20:52:10.412322 1208 raft_consensus.cc:2272] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.412911 1208 ts_tablet_manager.cc:1916] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.415257 1205 ts_tablet_manager.cc:1929] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:10.416635 742 tablet_service.cc:1505] Processing CreateTablet for tablet ed1d007f1ca24b7291262708ddfca3c3 (DEFAULT_TABLE table=default.Table_e3603e6d524c40dea2a1a88b79d63a8e [id=7febb36ade704d57bdfd4211054277ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.417062 1205 log.cc:1199] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/2a0eb9ab79d9428997c6f66790d6b1a3
I20251214 20:52:10.417196 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed1d007f1ca24b7291262708ddfca3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.418172 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a142157acbcd44adb6099b4428103ee5) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.417764 1205 ts_tablet_manager.cc:1950] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:10.420269 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (table default.Table_efa9fa90866441c8a4d8a14d14bb4bdd [id=0a848dd5f6474712b2eba70add3743dc]) successfully deleted
I20251214 20:52:10.420981 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44f51e11041841e98b9a15d380226e4e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:10.421574 1203 tablet_bootstrap.cc:654] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:10.422206 991 leader_election.cc:343] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.422616 1205 tablet_replica.cc:333] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:10.422705 991 leader_election.cc:343] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.424167 1205 raft_consensus.cc:2243] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.424401 1205 raft_consensus.cc:2272] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.424413 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47876b7d13f6455fa367933990ad2f6e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:10.424487 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6381cbf05a8c4f84b71f229497fd28ef" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:10.424898 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6381cbf05a8c4f84b71f229497fd28ef" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
W20251214 20:52:10.425539 712 leader_election.cc:343] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: Tablet not RUNNING: STOPPED
I20251214 20:52:10.425675 712 leader_election.cc:304] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:10.426083 891 tablet_service.cc:1505] Processing CreateTablet for tablet 956e4b4493f24109ab1251dde0a4cfc0 (DEFAULT_TABLE table=default.Table_0051db617c204791a9ed8d4795676c79 [id=0453c668e17b4c1ba55df9ff8fb1fce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.427796 1202 tablet_bootstrap.cc:492] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:10.427649 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_209aeff6e9694fc6a42598018aba1dc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:10.428162 1202 ts_tablet_manager.cc:1403] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.165s user 0.003s sys 0.011s
W20251214 20:52:10.428359 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_209aeff6e9694fc6a42598018aba1dc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.428262 518 catalog_manager.cc:5654] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:10.429620 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_4e3ce0b833374349ab211134c2709e81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:10.429682 1202 raft_consensus.cc:359] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.429879 1202 raft_consensus.cc:385] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.429940 1202 raft_consensus.cc:740] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
W20251214 20:52:10.430089 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e3ce0b833374349ab211134c2709e81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.430295 1202 consensus_queue.cc:260] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.431797 1202 ts_tablet_manager.cc:1434] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:10.432621 1205 ts_tablet_manager.cc:1916] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.436133 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 956e4b4493f24109ab1251dde0a4cfc0 (DEFAULT_TABLE table=default.Table_0051db617c204791a9ed8d4795676c79 [id=0453c668e17b4c1ba55df9ff8fb1fce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.438889 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 956e4b4493f24109ab1251dde0a4cfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.439574 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_3c3f2d9e5c2b47ed8038a7402612b093" } modify_external_catalogs: true
I20251214 20:52:10.439831 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_3c3f2d9e5c2b47ed8038a7402612b093" } modify_external_catalogs: true
I20251214 20:52:10.451367 1208 ts_tablet_manager.cc:1929] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.453522 1208 log.cc:1199] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/47876b7d13f6455fa367933990ad2f6e
I20251214 20:52:10.456542 1279 raft_consensus.cc:2749] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:10.454901 1208 ts_tablet_manager.cc:1950] T 47876b7d13f6455fa367933990ad2f6e P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:10.458024 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 47876b7d13f6455fa367933990ad2f6e (table default.Table_fcdbff382a58458bb3b9912ed28909bf [id=13be9c7a389b4137a6bcdf57d756bfa7]) successfully deleted
I20251214 20:52:10.459052 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 0c344296be0549e88d4811103eb6fee5
I20251214 20:52:10.460501 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c344296be0549e88d4811103eb6fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.460565 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c344296be0549e88d4811103eb6fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:10.461320 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 35101a6997e24040994661756d9649d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35101a6997e24040994661756d9649d4 already in progress: creating tablet
W20251214 20:52:10.461544 712 leader_election.cc:343] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:10.461858 712 leader_election.cc:343] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:10.462108 512 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 810ba69dda7f4631b1432bd99029609b because the tablet was not found. No further retry: Not found: Tablet not found: 810ba69dda7f4631b1432bd99029609b
I20251214 20:52:10.463451 1205 ts_tablet_manager.cc:1929] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.463531 1181 webserver.cc:492] Webserver started at http://127.0.104.60:40473/ using document root <none> and password file <none>
W20251214 20:52:10.464664 512 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 810ba69dda7f4631b1432bd99029609b because the tablet was not found. No further retry: Not found: Tablet not found: 810ba69dda7f4631b1432bd99029609b
I20251214 20:52:10.464665 1181 fs_manager.cc:362] Metadata directory not provided
I20251214 20:52:10.464810 1181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20251214 20:52:10.465317 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a0eb9ab79d9428997c6f66790d6b1a3 already in progress: creating tablet
W20251214 20:52:10.465909 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 0c344296be0549e88d4811103eb6fee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c344296be0549e88d4811103eb6fee5 already in progress: creating tablet
I20251214 20:52:10.467155 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_017c57bf3637493fb9ccb5976322bfe7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.468128 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_017c57bf3637493fb9ccb5976322bfe7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.469266 1205 log.cc:1199] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ac19d32a31e7405b9f16566215d6b2cd
I20251214 20:52:10.469662 1205 ts_tablet_manager.cc:1950] T ac19d32a31e7405b9f16566215d6b2cd P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:10.469767 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_af3089686b244e669ac592b91a9e99e6" } new_table_name: "default.Table_0173f37106d54db18ef3975bc6b0d185" modify_external_catalogs: true
I20251214 20:52:10.471771 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_c16ccd85e43049d69fa14c92e073791c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:10.472239 1205 tablet_replica.cc:333] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:10.472440 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c16ccd85e43049d69fa14c92e073791c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.472494 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ac19d32a31e7405b9f16566215d6b2cd (table default.Table_8090a0016b2248f6b8fa395e4f676734 [id=ac9c660a341b4fe6abedd5343196a30a]) successfully deleted
W20251214 20:52:10.473448 513 catalog_manager.cc:5058] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): alter failed for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (table default.Table_efa9fa90866441c8a4d8a14d14bb4bdd [id=0a848dd5f6474712b2eba70add3743dc]),no further retry: Not found: Tablet not found: 2a0eb9ab79d9428997c6f66790d6b1a3
I20251214 20:52:10.473835 1205 raft_consensus.cc:2243] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.475042 1205 raft_consensus.cc:2272] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.475263 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_4e46507057c94aa7a347491bf2e482ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.475886 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e46507057c94aa7a347491bf2e482ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.475943 1205 ts_tablet_manager.cc:1916] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.477178 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_26a97fa4596e4f769036f616c0d377df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.477708 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26a97fa4596e4f769036f616c0d377df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.479219 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_22df18c031ab4778ae90eb3580e9b8b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.479691 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22df18c031ab4778ae90eb3580e9b8b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.481812 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d7cbf1ffbce44772951752ecbb001da6" } new_table_name: "default.Table_d6e5176452664864914a6f76f32af5ac" modify_external_catalogs: true
I20251214 20:52:10.481833 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_dd29a2a18b4449ee8e686081ba7f5d25" } new_table_name: "default.Table_28368c3015e84594800ff8a110b0b4c3" modify_external_catalogs: true
W20251214 20:52:10.486181 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251214 20:52:10.487972 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_ce0f7774a3c249cfb10eafbbf0156e68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.488996 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce0f7774a3c249cfb10eafbbf0156e68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.490536 1211 raft_consensus.cc:493] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:10.490754 1211 raft_consensus.cc:515] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.491583 1211 leader_election.cc:290] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:10.491942 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_a3c01a9acc334c4cba07c4646084e233"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.492457 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3c01a9acc334c4cba07c4646084e233 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.484936 1205 ts_tablet_manager.cc:1929] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.493148 1205 log.cc:1199] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/1e95f3d09dac4c2bb838d73f65df64a6
I20251214 20:52:10.493474 1205 ts_tablet_manager.cc:1950] T 1e95f3d09dac4c2bb838d73f65df64a6 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:10.494109 1181 fs_manager.cc:714] Time spent opening directory manager: real 0.027s user 0.000s sys 0.010s
I20251214 20:52:10.494220 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cc54be0bc6d4f259967fbe0a5023f82" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:10.494525 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 1e95f3d09dac4c2bb838d73f65df64a6 (table default.Table_f46dedc9f5c44386abeb1d210d19b59e [id=53fd8efd1a1f4b7dac3c57c8e66bdc61]) successfully deleted
W20251214 20:52:10.494869 989 leader_election.cc:343] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.495833 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cc54be0bc6d4f259967fbe0a5023f82" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:10.496084 1205 tablet_replica.cc:333] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:10.496104 915 raft_consensus.cc:2468] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:10.496271 1205 raft_consensus.cc:2243] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.496455 1205 raft_consensus.cc:2272] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.496722 991 leader_election.cc:304] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:10.499061 1211 raft_consensus.cc:2804] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:10.499210 1211 raft_consensus.cc:493] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:10.499312 1211 raft_consensus.cc:3060] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.502108 1211 raft_consensus.cc:515] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.503445 1211 leader_election.cc:290] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:10.503780 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cc54be0bc6d4f259967fbe0a5023f82" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:10.504361 1205 ts_tablet_manager.cc:1916] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.504330 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cc54be0bc6d4f259967fbe0a5023f82" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:10.504575 915 raft_consensus.cc:3060] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:10.506029 1283 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20251214 20:52:10.507622 989 leader_election.cc:343] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.509420 915 raft_consensus.cc:2468] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:10.510104 991 leader_election.cc:304] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:10.510463 1211 raft_consensus.cc:2804] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:10.510612 1211 raft_consensus.cc:697] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:10.510913 1211 consensus_queue.cc:237] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.511801 1181 fs_manager.cc:730] Time spent opening block manager: real 0.015s user 0.000s sys 0.006s
I20251214 20:52:10.513077 1181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal
uuid: "7c48b8eaf30c4e2caa94a51d087406d4"
format_stamp: "Formatted at 2025-12-14 20:52:05 on dist-test-slave-njhz"
I20251214 20:52:10.515374 1205 ts_tablet_manager.cc:1929] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.515600 1205 log.cc:1199] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/47876b7d13f6455fa367933990ad2f6e
I20251214 20:52:10.515759 1181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251214 20:52:10.516014 1205 ts_tablet_manager.cc:1950] T 47876b7d13f6455fa367933990ad2f6e P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:10.517066 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 47876b7d13f6455fa367933990ad2f6e (table default.Table_fcdbff382a58458bb3b9912ed28909bf [id=13be9c7a389b4137a6bcdf57d756bfa7]) successfully deleted
I20251214 20:52:10.517454 1205 tablet_replica.cc:333] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:10.517724 1205 raft_consensus.cc:2243] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.517951 1205 raft_consensus.cc:2272] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.518850 1205 ts_tablet_manager.cc:1916] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.521310 1202 tablet_bootstrap.cc:492] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:10.523912 1202 tablet_bootstrap.cc:654] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.526960 1206 tablet_bootstrap.cc:492] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.527210 1206 ts_tablet_manager.cc:1403] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.294s user 0.009s sys 0.018s
W20251214 20:52:10.527302 518 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fd0fa0dad974c98bf6f52c8cfe83035 consensus state has no leader
I20251214 20:52:10.528191 1206 raft_consensus.cc:359] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.528417 1206 raft_consensus.cc:385] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.528527 1206 raft_consensus.cc:740] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.528806 1206 consensus_queue.cc:260] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.529497 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_500b29027c11444893ae5b7a3123cb8b" } modify_external_catalogs: true
I20251214 20:52:10.529664 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 956e4b4493f24109ab1251dde0a4cfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.529735 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_500b29027c11444893ae5b7a3123cb8b" } modify_external_catalogs: true
W20251214 20:52:10.542501 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a0ca65d369a45958c6dac539964114b consensus state has no leader
W20251214 20:52:10.543206 523 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8140dad1fab142d98b930b53ae5a64c2 consensus state has no leader
I20251214 20:52:10.544238 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_834f1d323b2b4785a85f667e5c14e739" } modify_external_catalogs: true
I20251214 20:52:10.544489 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_834f1d323b2b4785a85f667e5c14e739" } modify_external_catalogs: true
I20251214 20:52:10.544950 891 tablet_service.cc:1505] Processing CreateTablet for tablet dc442726ec0c456290ecf2892d0df957 (DEFAULT_TABLE table=default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.545349 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc442726ec0c456290ecf2892d0df957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.545379 1205 ts_tablet_manager.cc:1929] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.545521 1205 log.cc:1199] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/4a350ae5742b40e3926a82c692933839
I20251214 20:52:10.546459 1205 ts_tablet_manager.cc:1950] T 4a350ae5742b40e3926a82c692933839 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:10.549827 892 tablet_service.cc:1505] Processing CreateTablet for tablet 118dd44151a34ba3ae58f9fe1d3790d4 (DEFAULT_TABLE table=default.Table_e8acf2f630434ce3b7f06a4a01e4a2a3 [id=ea94fa24050345f98516132203c9704c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.550241 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118dd44151a34ba3ae58f9fe1d3790d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.548370 1031 tablet_service.cc:1505] Processing CreateTablet for tablet dc442726ec0c456290ecf2892d0df957 (DEFAULT_TABLE table=default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.551247 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc442726ec0c456290ecf2892d0df957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.557989 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 4a350ae5742b40e3926a82c692933839 (table default.Table_6d8ae2f7aafa4f28a7430e353d201b63 [id=2a868a8e093247baa304d4fe1422c59a]) successfully deleted
I20251214 20:52:10.558298 518 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet ecb69e385bf34709b5d954de4b660087
W20251214 20:52:10.558543 513 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 810ba69dda7f4631b1432bd99029609b because the tablet was not found. No further retry: Not found: Tablet not found: 810ba69dda7f4631b1432bd99029609b
I20251214 20:52:10.559698 1205 tablet_replica.cc:333] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:10.559729 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet ecb69e385bf34709b5d954de4b660087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.559937 892 tablet_service.cc:1552] Processing DeleteTablet for tablet ecb69e385bf34709b5d954de4b660087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.560227 1205 raft_consensus.cc:2243] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:10.560551 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ecb69e385bf34709b5d954de4b660087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb69e385bf34709b5d954de4b660087 already in progress: creating tablet
I20251214 20:52:10.560825 1205 raft_consensus.cc:2272] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.561358 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c344296be0549e88d4811103eb6fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.561604 525 catalog_manager.cc:5654] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:10.562302 525 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:10.562642 756 tablet_service.cc:1505] Processing CreateTablet for tablet dc442726ec0c456290ecf2892d0df957 (DEFAULT_TABLE table=default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.563002 747 tablet_service.cc:1552] Processing DeleteTablet for tablet ecb69e385bf34709b5d954de4b660087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.563789 1206 ts_tablet_manager.cc:1434] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:10.564167 1206 tablet_bootstrap.cc:492] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:10.564443 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc442726ec0c456290ecf2892d0df957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.565490 742 tablet_service.cc:1505] Processing CreateTablet for tablet 956e4b4493f24109ab1251dde0a4cfc0 (DEFAULT_TABLE table=default.Table_0051db617c204791a9ed8d4795676c79 [id=0453c668e17b4c1ba55df9ff8fb1fce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.565891 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 956e4b4493f24109ab1251dde0a4cfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.569797 523 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet c17126f75b6241c5833269d20774c439
I20251214 20:52:10.571014 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet c17126f75b6241c5833269d20774c439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.571869 892 tablet_service.cc:1552] Processing DeleteTablet for tablet c17126f75b6241c5833269d20774c439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.572419 747 tablet_service.cc:1552] Processing DeleteTablet for tablet c17126f75b6241c5833269d20774c439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
W20251214 20:52:10.572613 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c17126f75b6241c5833269d20774c439 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c17126f75b6241c5833269d20774c439 already in progress: creating tablet
I20251214 20:52:10.575680 1205 ts_tablet_manager.cc:1916] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.576042 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_934b5fc4d4004b68b7a77ec2cfd47f8c" } modify_external_catalogs: true
I20251214 20:52:10.576349 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_934b5fc4d4004b68b7a77ec2cfd47f8c" } modify_external_catalogs: true
I20251214 20:52:10.577106 1203 tablet_bootstrap.cc:492] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:10.577606 1203 ts_tablet_manager.cc:1403] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.193s user 0.026s sys 0.074s
I20251214 20:52:10.577979 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_035a23623ce242298ab89c51024d7a09" } modify_external_catalogs: true
I20251214 20:52:10.578593 1203 raft_consensus.cc:359] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.578732 1206 tablet_bootstrap.cc:654] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.578804 1203 raft_consensus.cc:385] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.578891 1203 raft_consensus.cc:740] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.579190 1203 consensus_queue.cc:260] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.580938 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 118dd44151a34ba3ae58f9fe1d3790d4 (DEFAULT_TABLE table=default.Table_e8acf2f630434ce3b7f06a4a01e4a2a3 [id=ea94fa24050345f98516132203c9704c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.581051 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_035a23623ce242298ab89c51024d7a09" } modify_external_catalogs: true
I20251214 20:52:10.579495 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_274bf0a7631e4c3c8144694db10a5de9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.581790 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_274bf0a7631e4c3c8144694db10a5de9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.581923 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118dd44151a34ba3ae58f9fe1d3790d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.583627 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_0051db617c204791a9ed8d4795676c79" } new_table_name: "default.Table_9447e701a9d0498f9c179a48dfb633b5" modify_external_catalogs: true
I20251214 20:52:10.578362 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ce0f7774a3c249cfb10eafbbf0156e68" } new_table_name: "default.Table_a2a64eab2d9a493da6fee1051eadd37d" modify_external_catalogs: true
I20251214 20:52:10.585222 1203 ts_tablet_manager.cc:1434] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251214 20:52:10.589079 1203 tablet_bootstrap.cc:492] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:10.591109 1203 tablet_bootstrap.cc:654] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.592877 1207 ts_tablet_manager.cc:1929] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.595093 892 tablet_service.cc:1505] Processing CreateTablet for tablet 376e5efede87410eb9f95f36bedebf9d (DEFAULT_TABLE table=default.Table_48c1e24ff9664a38a72467f0543209b7 [id=0b01345b8b0b44c0b59bced8e8b89a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.595741 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 376e5efede87410eb9f95f36bedebf9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.599327 1205 ts_tablet_manager.cc:1929] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.599815 1205 log.cc:1199] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/6e4c1451b8ea4b6f8c934633f67212b0
I20251214 20:52:10.600416 1205 ts_tablet_manager.cc:1950] T 6e4c1451b8ea4b6f8c934633f67212b0 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:10.600508 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 23ecf83d935c49f9b304a3355b0cc55d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.601420 1203 tablet_bootstrap.cc:492] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:10.601588 1203 ts_tablet_manager.cc:1403] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20251214 20:52:10.602653 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 6e4c1451b8ea4b6f8c934633f67212b0 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]) successfully deleted
I20251214 20:52:10.602816 1203 raft_consensus.cc:359] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.603163 1203 raft_consensus.cc:385] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.603183 1202 tablet_bootstrap.cc:492] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:10.603271 1203 raft_consensus.cc:740] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.603328 1202 ts_tablet_manager.cc:1403] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.001s
I20251214 20:52:10.603544 1203 consensus_queue.cc:260] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:10.604133 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 0c344296be0549e88d4811103eb6fee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c344296be0549e88d4811103eb6fee5 already in progress: creating tablet
I20251214 20:52:10.604573 1202 raft_consensus.cc:359] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.604815 1202 raft_consensus.cc:385] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.604950 1202 raft_consensus.cc:740] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
W20251214 20:52:10.605113 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ecb69e385bf34709b5d954de4b660087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb69e385bf34709b5d954de4b660087 already in progress: creating tablet
I20251214 20:52:10.605374 1202 consensus_queue.cc:260] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.606263 523 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet f2b66900d6ea42aa849c24acbbe23c40
W20251214 20:52:10.606923 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c17126f75b6241c5833269d20774c439 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c17126f75b6241c5833269d20774c439 already in progress: creating tablet
I20251214 20:52:10.607493 1205 tablet_replica.cc:333] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:10.607740 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet f2b66900d6ea42aa849c24acbbe23c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.608187 1205 raft_consensus.cc:2243] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.608659 1205 raft_consensus.cc:2272] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.609226 1202 ts_tablet_manager.cc:1434] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251214 20:52:10.609843 1202 tablet_bootstrap.cc:492] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:10.609174 1203 ts_tablet_manager.cc:1434] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251214 20:52:10.612538 1203 tablet_bootstrap.cc:492] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:10.612912 1202 tablet_bootstrap.cc:654] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.614356 1203 tablet_bootstrap.cc:654] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.614514 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 69d44f933f494668b2881396e02669ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.614843 1208 tablet_replica.cc:333] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:10.615065 1208 raft_consensus.cc:2243] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.615293 1208 raft_consensus.cc:2272] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.615993 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 35101a6997e24040994661756d9649d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.616632 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 376e5efede87410eb9f95f36bedebf9d (DEFAULT_TABLE table=default.Table_48c1e24ff9664a38a72467f0543209b7 [id=0b01345b8b0b44c0b59bced8e8b89a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.617110 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 376e5efede87410eb9f95f36bedebf9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.617831 1208 ts_tablet_manager.cc:1916] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.624670 895 tablet_service.cc:1505] Processing CreateTablet for tablet 84a0237e4ee34f94821d8647a400575b (DEFAULT_TABLE table=default.Table_3c8ed47b57e546f59c9d863f953a17ec [id=fa79ab98ae4f405d830ed263d3c0b3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.625093 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a0237e4ee34f94821d8647a400575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.634831 1208 ts_tablet_manager.cc:1929] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.593375 1217 raft_consensus.cc:697] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:10.594826 756 tablet_service.cc:1505] Processing CreateTablet for tablet 118dd44151a34ba3ae58f9fe1d3790d4 (DEFAULT_TABLE table=default.Table_e8acf2f630434ce3b7f06a4a01e4a2a3 [id=ea94fa24050345f98516132203c9704c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.635306 1207 log.cc:1199] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/31f74b0b72b74085b31c16caca2efa08
I20251214 20:52:10.635775 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118dd44151a34ba3ae58f9fe1d3790d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.635694 1217 consensus_queue.cc:237] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.638440 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 12b5f4326aa0413eb303c132ba21decd (DEFAULT_TABLE table=default.Table_6061b180a35048c4a55552330aef8b00 [id=b5ef4fb9aeba4c3fb423322b5f1d0e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.609231 747 tablet_service.cc:1552] Processing DeleteTablet for tablet f2b66900d6ea42aa849c24acbbe23c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.624584 745 tablet_service.cc:1505] Processing CreateTablet for tablet 376e5efede87410eb9f95f36bedebf9d (DEFAULT_TABLE table=default.Table_48c1e24ff9664a38a72467f0543209b7 [id=0b01345b8b0b44c0b59bced8e8b89a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.638932 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b5f4326aa0413eb303c132ba21decd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.639528 747 tablet_service.cc:1505] Processing CreateTablet for tablet 84a0237e4ee34f94821d8647a400575b (DEFAULT_TABLE table=default.Table_3c8ed47b57e546f59c9d863f953a17ec [id=fa79ab98ae4f405d830ed263d3c0b3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.639962 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a0237e4ee34f94821d8647a400575b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:10.642894 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 956e4b4493f24109ab1251dde0a4cfc0 consensus state has no leader
I20251214 20:52:10.643860 518 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 6381cbf05a8c4f84b71f229497fd28ef
I20251214 20:52:10.644606 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 84a0237e4ee34f94821d8647a400575b (DEFAULT_TABLE table=default.Table_3c8ed47b57e546f59c9d863f953a17ec [id=fa79ab98ae4f405d830ed263d3c0b3c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.645267 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a0237e4ee34f94821d8647a400575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.645573 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 376e5efede87410eb9f95f36bedebf9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.646008 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 6381cbf05a8c4f84b71f229497fd28ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:10.649070 522 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2842f1dc0c374d24946f90dac1cf8aa5 has no consensus state
I20251214 20:52:10.649691 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 6381cbf05a8c4f84b71f229497fd28ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.652658 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_ca875b1006554ba6845c0cf23671a8da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.653553 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca875b1006554ba6845c0cf23671a8da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.655202 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_d553ca568da242899d768f9848baba3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.655840 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d553ca568da242899d768f9848baba3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.656109 892 tablet_service.cc:1552] Processing DeleteTablet for tablet f2b66900d6ea42aa849c24acbbe23c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.657590 520 master_service.cc:954] ReplaceTablet: received request to replace tablet ebedf28e17a741169e5490822a2d95df from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.658730 525 master_service.cc:954] ReplaceTablet: received request to replace tablet f44e7223f79041ec9d66d0a1bdcd4ff5 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.658735 751 tablet_service.cc:1505] Processing CreateTablet for tablet 12b5f4326aa0413eb303c132ba21decd (DEFAULT_TABLE table=default.Table_6061b180a35048c4a55552330aef8b00 [id=b5ef4fb9aeba4c3fb423322b5f1d0e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.659691 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 810ba69dda7f4631b1432bd99029609b (DEFAULT_TABLE table=default.Table_850408f19d2848309b88410e331f0798 [id=c544a159297d49c8be1c9a0eff9986cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.660172 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b5f4326aa0413eb303c132ba21decd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.660187 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810ba69dda7f4631b1432bd99029609b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.664222 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_28368c3015e84594800ff8a110b0b4c3" } modify_external_catalogs: true
I20251214 20:52:10.664559 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_28368c3015e84594800ff8a110b0b4c3" } modify_external_catalogs: true
I20251214 20:52:10.665253 749 tablet_service.cc:1505] Processing CreateTablet for tablet 810ba69dda7f4631b1432bd99029609b (DEFAULT_TABLE table=default.Table_850408f19d2848309b88410e331f0798 [id=c544a159297d49c8be1c9a0eff9986cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.665796 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810ba69dda7f4631b1432bd99029609b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.666646 1217 raft_consensus.cc:493] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:10.666851 1217 raft_consensus.cc:515] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.668108 1217 leader_election.cc:290] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:10.669747 713 leader_election.cc:343] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: ac19d32a31e7405b9f16566215d6b2cd
I20251214 20:52:10.672461 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac19d32a31e7405b9f16566215d6b2cd" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:10.673085 712 leader_election.cc:343] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.673280 712 leader_election.cc:304] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:10.673695 1217 raft_consensus.cc:2749] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:10.675172 892 tablet_service.cc:1505] Processing CreateTablet for tablet 810ba69dda7f4631b1432bd99029609b (DEFAULT_TABLE table=default.Table_850408f19d2848309b88410e331f0798 [id=c544a159297d49c8be1c9a0eff9986cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.675700 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810ba69dda7f4631b1432bd99029609b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.676060 1206 tablet_bootstrap.cc:492] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.676374 520 master_service.cc:954] ReplaceTablet: received request to replace tablet dc442726ec0c456290ecf2892d0df957 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.676734 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 5ba2d34a6d2645bd97fe85fed952692b from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.677874 1203 tablet_bootstrap.cc:492] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:10.678090 1203 ts_tablet_manager.cc:1403] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.009s
I20251214 20:52:10.678414 1202 tablet_bootstrap.cc:492] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:10.678880 1203 raft_consensus.cc:359] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.679085 1203 raft_consensus.cc:385] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.679196 1203 raft_consensus.cc:740] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.679505 1206 ts_tablet_manager.cc:1403] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.115s user 0.000s sys 0.004s
I20251214 20:52:10.679715 1202 ts_tablet_manager.cc:1403] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20251214 20:52:10.681746 1202 raft_consensus.cc:359] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.684554 1206 raft_consensus.cc:359] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.684912 1206 raft_consensus.cc:385] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.685025 1206 raft_consensus.cc:740] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.685304 1203 consensus_queue.cc:260] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.685706 1206 consensus_queue.cc:260] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.685658 1035 tablet_service.cc:1505] Processing CreateTablet for tablet b0e805cc899347a083d4d549ad68e2f9 (DEFAULT_TABLE table=default.Table_46ba3258cb8d46b2b0f28e6b98eb8603 [id=ae6d8ad2d21d4208a1c19435e108f95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.686340 1203 ts_tablet_manager.cc:1434] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251214 20:52:10.687613 1206 ts_tablet_manager.cc:1434] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:10.687932 1206 tablet_bootstrap.cc:492] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:10.686316 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0e805cc899347a083d4d549ad68e2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.688251 1203 tablet_bootstrap.cc:492] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:10.684813 890 tablet_service.cc:1505] Processing CreateTablet for tablet b0e805cc899347a083d4d549ad68e2f9 (DEFAULT_TABLE table=default.Table_46ba3258cb8d46b2b0f28e6b98eb8603 [id=ae6d8ad2d21d4208a1c19435e108f95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.690343 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0e805cc899347a083d4d549ad68e2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.691195 1206 tablet_bootstrap.cc:654] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.692407 893 tablet_service.cc:1505] Processing CreateTablet for tablet 569a02142c71427d91868b02695ae12b (DEFAULT_TABLE table=default.Table_ad4bec26fc854d67b2ec37f504851a01 [id=17c89c03be10405f834504b803e594c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.692950 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569a02142c71427d91868b02695ae12b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.697486 749 tablet_service.cc:1505] Processing CreateTablet for tablet b0e805cc899347a083d4d549ad68e2f9 (DEFAULT_TABLE table=default.Table_46ba3258cb8d46b2b0f28e6b98eb8603 [id=ae6d8ad2d21d4208a1c19435e108f95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.698110 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0e805cc899347a083d4d549ad68e2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.699478 1206 tablet_bootstrap.cc:492] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.702353 523 master_service.cc:954] ReplaceTablet: received request to replace tablet ebedf28e17a741169e5490822a2d95df from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.703318 1206 ts_tablet_manager.cc:1403] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20251214 20:52:10.703560 745 tablet_service.cc:1505] Processing CreateTablet for tablet 569a02142c71427d91868b02695ae12b (DEFAULT_TABLE table=default.Table_ad4bec26fc854d67b2ec37f504851a01 [id=17c89c03be10405f834504b803e594c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.704070 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569a02142c71427d91868b02695ae12b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.703665 742 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19d32a31e7405b9f16566215d6b2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.704388 1206 raft_consensus.cc:359] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.704595 1206 raft_consensus.cc:385] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.704691 1206 raft_consensus.cc:740] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.706125 1206 consensus_queue.cc:260] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.706983 1206 ts_tablet_manager.cc:1434] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251214 20:52:10.709139 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 6381cbf05a8c4f84b71f229497fd28ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.709722 890 tablet_service.cc:1505] Processing CreateTablet for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (DEFAULT_TABLE table=default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.710515 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13f93bb5e5f4b59bcf9f4bfed070f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.635171 1208 log.cc:1199] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/69d44f933f494668b2881396e02669ab
I20251214 20:52:10.711171 1208 ts_tablet_manager.cc:1950] T 69d44f933f494668b2881396e02669ab P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:10.712491 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 69d44f933f494668b2881396e02669ab (table default.Table_ea06b0c0cb094fba91bd4b311ba90194 [id=5040c9d5f4d5430dbff0ab01f2c71508]) successfully deleted
W20251214 20:52:10.713690 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet f2b66900d6ea42aa849c24acbbe23c40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2b66900d6ea42aa849c24acbbe23c40 already in progress: creating tablet
I20251214 20:52:10.714066 1203 tablet_bootstrap.cc:654] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.635775 1207 ts_tablet_manager.cc:1950] T 31f74b0b72b74085b31c16caca2efa08 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:10.716868 519 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:10.717408 1207 tablet_replica.cc:333] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:10.717633 1207 raft_consensus.cc:2243] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:10.719691 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 31f74b0b72b74085b31c16caca2efa08 (table default.Table_bd2058c3daa94a0cbabdda39dbf5ec2b [id=fe9038b8f14d46e8aadc6bf34505b684]) successfully deleted
I20251214 20:52:10.720595 1207 raft_consensus.cc:2272] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.721037 525 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 0a0ca65d369a45958c6dac539964114b
I20251214 20:52:10.721228 1207 ts_tablet_manager.cc:1916] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.722220 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a0ca65d369a45958c6dac539964114b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.722563 1206 tablet_bootstrap.cc:492] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:10.723742 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a0ca65d369a45958c6dac539964114b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.725204 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_03651b0344ca49b688101a958c2090e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.725979 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03651b0344ca49b688101a958c2090e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.727816 1206 tablet_bootstrap.cc:654] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.728698 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet dc442726ec0c456290ecf2892d0df957
I20251214 20:52:10.729228 520 catalog_manager.cc:6724] ReplaceTablet: tablet dc442726ec0c456290ecf2892d0df957 deleted and replaced by tablet 6bd6856b800247de896f16eee66f2827
I20251214 20:52:10.729598 526 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 5ba2d34a6d2645bd97fe85fed952692b
I20251214 20:52:10.729941 526 catalog_manager.cc:6724] ReplaceTablet: tablet 5ba2d34a6d2645bd97fe85fed952692b deleted and replaced by tablet f0d661a1bb034053a4634ff537eafc7d
I20251214 20:52:10.730831 745 tablet_service.cc:1552] Processing DeleteTablet for tablet dc442726ec0c456290ecf2892d0df957 with delete_type TABLET_DATA_DELETED (replaced by tablet 6bd6856b800247de896f16eee66f2827) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.733410 745 tablet_service.cc:1505] Processing CreateTablet for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (DEFAULT_TABLE table=default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.734408 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13f93bb5e5f4b59bcf9f4bfed070f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.734885 751 tablet_service.cc:1505] Processing CreateTablet for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 (DEFAULT_TABLE table=default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.734941 749 tablet_service.cc:1505] Processing CreateTablet for tablet 63ef97aeeaa94f4da683fbd133b05bd1 (DEFAULT_TABLE table=default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.735545 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ba2d34a6d2645bd97fe85fed952692b with delete_type TABLET_DATA_DELETED (replaced by tablet f0d661a1bb034053a4634ff537eafc7d) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.737638 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2ee5cc95304fc1a80ed4d31838ac38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.737541 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_9ceff170cc7f4bac8ae6eaf248c5e1e9" } new_table_name: "default.Table_4efe28833e3d4ad8b4e1bb8966338a62" modify_external_catalogs: true
I20251214 20:52:10.738332 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 569a02142c71427d91868b02695ae12b (DEFAULT_TABLE table=default.Table_ad4bec26fc854d67b2ec37f504851a01 [id=17c89c03be10405f834504b803e594c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.738802 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569a02142c71427d91868b02695ae12b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.739071 1202 raft_consensus.cc:385] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.739243 1202 raft_consensus.cc:740] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.739674 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_847906e708bd41e9b9507efc8f52347f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:10.739542 1202 consensus_queue.cc:260] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:10.740214 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_847906e708bd41e9b9507efc8f52347f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.742106 891 tablet_service.cc:1505] Processing CreateTablet for tablet 12b5f4326aa0413eb303c132ba21decd (DEFAULT_TABLE table=default.Table_6061b180a35048c4a55552330aef8b00 [id=b5ef4fb9aeba4c3fb423322b5f1d0e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.742305 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_d653f9c533784976ac955f90dc39eb56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.742790 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d653f9c533784976ac955f90dc39eb56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.743443 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet dc442726ec0c456290ecf2892d0df957 with delete_type TABLET_DATA_DELETED (replaced by tablet 6bd6856b800247de896f16eee66f2827) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.743778 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a0ca65d369a45958c6dac539964114b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.743980 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_cb89b84fb9ec4a0d89a3fd65d538725e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.744410 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb89b84fb9ec4a0d89a3fd65d538725e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.745586 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_18dc9de393304efcb3ec4eb6be858624"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.747316 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18dc9de393304efcb3ec4eb6be858624 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.747339 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ba2d34a6d2645bd97fe85fed952692b with delete_type TABLET_DATA_DELETED (replaced by tablet f0d661a1bb034053a4634ff537eafc7d) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:10.748664 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 (DEFAULT_TABLE table=default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.750789 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2ee5cc95304fc1a80ed4d31838ac38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.751398 1027 tablet_service.cc:1505] Processing CreateTablet for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (DEFAULT_TABLE table=default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.752214 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63ef97aeeaa94f4da683fbd133b05bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.752161 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_cebd52de786f4a79b9228aebdd8d69fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.752646 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cebd52de786f4a79b9228aebdd8d69fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.752873 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13f93bb5e5f4b59bcf9f4bfed070f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.753762 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_48c1e24ff9664a38a72467f0543209b7" } new_table_name: "default.Table_2dab958dcddb48e28bfc246163f539ae" modify_external_catalogs: true
I20251214 20:52:10.755178 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 63ef97aeeaa94f4da683fbd133b05bd1 (DEFAULT_TABLE table=default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.755249 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4e3ce0b833374349ab211134c2709e81" } new_table_name: "default.Table_1d198a3975c24b84aaeb82c266c746b8" modify_external_catalogs: true
I20251214 20:52:10.755740 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63ef97aeeaa94f4da683fbd133b05bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.760043 1207 ts_tablet_manager.cc:1929] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.760401 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_cf3c60b13d0542f4ae62c69c088edafb" } modify_external_catalogs: true
I20251214 20:52:10.760564 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_cf3c60b13d0542f4ae62c69c088edafb" } modify_external_catalogs: true
I20251214 20:52:10.761677 1207 log.cc:1199] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/47876b7d13f6455fa367933990ad2f6e
I20251214 20:52:10.762964 1207 ts_tablet_manager.cc:1950] T 47876b7d13f6455fa367933990ad2f6e P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:10.766023 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_cf7dd9af89cc4f06adec7de4838b9a32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:10.766518 1181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20251214 20:52:10.766812 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf7dd9af89cc4f06adec7de4838b9a32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.767670 1206 tablet_bootstrap.cc:492] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.767853 1206 ts_tablet_manager.cc:1403] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20251214 20:52:10.768700 1206 raft_consensus.cc:359] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.768890 1206 raft_consensus.cc:385] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.768962 1206 raft_consensus.cc:740] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.769248 1206 consensus_queue.cc:260] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.773011 1206 ts_tablet_manager.cc:1434] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20251214 20:52:10.773921 1205 ts_tablet_manager.cc:1916] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.773777 1206 tablet_bootstrap.cc:492] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:10.776486 1207 tablet_replica.cc:333] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:10.776791 1207 raft_consensus.cc:2243] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:10.776803 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b consensus state has no leader
I20251214 20:52:10.777031 1207 raft_consensus.cc:2272] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.777618 1207 ts_tablet_manager.cc:1916] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.777966 1206 tablet_bootstrap.cc:654] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.778613 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_22df18c031ab4778ae90eb3580e9b8b7" } modify_external_catalogs: true
I20251214 20:52:10.778791 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_22df18c031ab4778ae90eb3580e9b8b7" } modify_external_catalogs: true
I20251214 20:52:10.782128 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 47876b7d13f6455fa367933990ad2f6e (table default.Table_fcdbff382a58458bb3b9912ed28909bf [id=13be9c7a389b4137a6bcdf57d756bfa7]) successfully deleted
W20251214 20:52:10.782557 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 35101a6997e24040994661756d9649d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35101a6997e24040994661756d9649d4 already in progress: creating tablet
W20251214 20:52:10.782869 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 810ba69dda7f4631b1432bd99029609b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 810ba69dda7f4631b1432bd99029609b already in progress: creating tablet
I20251214 20:52:10.783591 745 tablet_service.cc:1552] Processing DeleteTablet for tablet ba2ed363bbab4f1f865b183297c98daa with delete_type TABLET_DATA_DELETED (replaced by tablet 569a02142c71427d91868b02695ae12b) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.787665 1207 ts_tablet_manager.cc:1929] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:10.787889 1206 tablet_bootstrap.cc:492] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.788024 1207 log.cc:1199] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/2a0eb9ab79d9428997c6f66790d6b1a3
I20251214 20:52:10.788046 1206 ts_tablet_manager.cc:1403] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20251214 20:52:10.788560 1207 ts_tablet_manager.cc:1950] T 2a0eb9ab79d9428997c6f66790d6b1a3 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:10.789002 1206 raft_consensus.cc:359] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.789207 1206 raft_consensus.cc:385] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.789320 1206 raft_consensus.cc:740] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.789862 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (table default.Table_efa9fa90866441c8a4d8a14d14bb4bdd [id=0a848dd5f6474712b2eba70add3743dc]) successfully deleted
I20251214 20:52:10.789921 1206 consensus_queue.cc:260] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.790081 1181 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251214 20:52:10.790436 1181 kserver.cc:163] Server-wide thread pool size limit: 3276
W20251214 20:52:10.791419 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 810ba69dda7f4631b1432bd99029609b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 810ba69dda7f4631b1432bd99029609b already in progress: creating tablet
I20251214 20:52:10.791833 1206 ts_tablet_manager.cc:1434] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:10.792448 1206 tablet_bootstrap.cc:492] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:10.792544 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 0c344296be0549e88d4811103eb6fee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c344296be0549e88d4811103eb6fee5 already in progress: creating tablet
W20251214 20:52:10.801514 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ecb69e385bf34709b5d954de4b660087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb69e385bf34709b5d954de4b660087 already in progress: creating tablet
W20251214 20:52:10.802052 521 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 376e5efede87410eb9f95f36bedebf9d consensus state has no leader
W20251214 20:52:10.802479 523 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8faa7fd7315f485c82aee38bcf4cd8fc has no consensus state
W20251214 20:52:10.802754 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c17126f75b6241c5833269d20774c439 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c17126f75b6241c5833269d20774c439 already in progress: creating tablet
I20251214 20:52:10.805914 745 tablet_service.cc:1505] Processing CreateTablet for tablet 63497e77baeb4d328be2421b5732a28a (DEFAULT_TABLE table=default.Table_d553ca568da242899d768f9848baba3d [id=376e4eabd04d4a53ab2772323b8c41b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.806397 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63497e77baeb4d328be2421b5732a28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.809064 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 144dee1a6d7344f2b79da6e79e67599e (DEFAULT_TABLE table=default.Table_c16ccd85e43049d69fa14c92e073791c [id=8cd62da3d57c4be4b5f10ace49d2e148]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.809548 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144dee1a6d7344f2b79da6e79e67599e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.811771 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 25974e5db555447baf5799775d15cd14 (DEFAULT_TABLE table=default.Table_209aeff6e9694fc6a42598018aba1dc1 [id=7dbd8a94ed0b497d8eba92091819fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.811882 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc (DEFAULT_TABLE table=default.Table_1d198a3975c24b84aaeb82c266c746b8 [id=bc62e2586be140efbc2bfb5a3b5baaa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.812183 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25974e5db555447baf5799775d15cd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.812397 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8faa7fd7315f485c82aee38bcf4cd8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.812855 1023 tablet_service.cc:1505] Processing CreateTablet for tablet ba50e32f7e724868bb891045a939e362 (DEFAULT_TABLE table=default.Table_348a96aca6644d08b3074727ff3df5a7 [id=c3e5006d02fc4b69987feb30ba66d069]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.813236 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba50e32f7e724868bb891045a939e362. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.818197 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_cebd52de786f4a79b9228aebdd8d69fb" } modify_external_catalogs: true
I20251214 20:52:10.818435 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_cebd52de786f4a79b9228aebdd8d69fb" } modify_external_catalogs: true
I20251214 20:52:10.818447 1206 tablet_bootstrap.cc:654] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.820772 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a0ca65d369a45958c6dac539964114b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.821524 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ba2d34a6d2645bd97fe85fed952692b with delete_type TABLET_DATA_DELETED (replaced by tablet f0d661a1bb034053a4634ff537eafc7d) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.823192 1217 raft_consensus.cc:493] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:10.823364 1217 raft_consensus.cc:515] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.823669 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b5f4326aa0413eb303c132ba21decd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.823989 1202 ts_tablet_manager.cc:1434] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
I20251214 20:52:10.824102 1217 leader_election.cc:290] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:10.824926 1207 tablet_replica.cc:333] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:10.825151 1207 raft_consensus.cc:2243] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:10.825259 893 tablet_service.cc:1505] Processing CreateTablet for tablet 023299180d074893847e84dcd9812751 (DEFAULT_TABLE table=default.Table_274bf0a7631e4c3c8144694db10a5de9 [id=c0729ac770394d7b87ef36e53f85c151]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.825428 749 tablet_service.cc:1505] Processing CreateTablet for tablet 6b4f582d8165439d86d9228c146dc41a (DEFAULT_TABLE table=default.Table_22df18c031ab4778ae90eb3580e9b8b7 [id=3f105cc6dd6944ceb9783a1ebe923d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.825613 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023299180d074893847e84dcd9812751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.825678 1207 pending_rounds.cc:70] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:10.825856 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b4f582d8165439d86d9228c146dc41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.825917 1207 pending_rounds.cc:77] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493693488840704 op_type: NO_OP noop_request { }
I20251214 20:52:10.826120 1207 raft_consensus.cc:2889] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:10.826246 1207 raft_consensus.cc:2272] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:10.826823 745 tablet_service.cc:1505] Processing CreateTablet for tablet 023299180d074893847e84dcd9812751 (DEFAULT_TABLE table=default.Table_274bf0a7631e4c3c8144694db10a5de9 [id=c0729ac770394d7b87ef36e53f85c151]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.827188 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023299180d074893847e84dcd9812751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.830322 889 tablet_service.cc:1505] Processing CreateTablet for tablet 63ef97aeeaa94f4da683fbd133b05bd1 (DEFAULT_TABLE table=default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.830715 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63ef97aeeaa94f4da683fbd133b05bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.831538 888 tablet_service.cc:1505] Processing CreateTablet for tablet 6b4f582d8165439d86d9228c146dc41a (DEFAULT_TABLE table=default.Table_22df18c031ab4778ae90eb3580e9b8b7 [id=3f105cc6dd6944ceb9783a1ebe923d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.831990 887 tablet_service.cc:1505] Processing CreateTablet for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 (DEFAULT_TABLE table=default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.832695 747 tablet_service.cc:1505] Processing CreateTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc (DEFAULT_TABLE table=default.Table_1d198a3975c24b84aaeb82c266c746b8 [id=bc62e2586be140efbc2bfb5a3b5baaa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.832799 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b4f582d8165439d86d9228c146dc41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.833129 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8faa7fd7315f485c82aee38bcf4cd8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.833067 751 tablet_service.cc:1505] Processing CreateTablet for tablet 144dee1a6d7344f2b79da6e79e67599e (DEFAULT_TABLE table=default.Table_c16ccd85e43049d69fa14c92e073791c [id=8cd62da3d57c4be4b5f10ace49d2e148]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.833372 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144dee1a6d7344f2b79da6e79e67599e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.833370 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a0ca65d369a45958c6dac539964114b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:10.833600 756 tablet_service.cc:1505] Processing CreateTablet for tablet 25974e5db555447baf5799775d15cd14 (DEFAULT_TABLE table=default.Table_209aeff6e9694fc6a42598018aba1dc1 [id=7dbd8a94ed0b497d8eba92091819fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.833945 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25974e5db555447baf5799775d15cd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.834582 894 tablet_service.cc:1505] Processing CreateTablet for tablet 63497e77baeb4d328be2421b5732a28a (DEFAULT_TABLE table=default.Table_d553ca568da242899d768f9848baba3d [id=376e4eabd04d4a53ab2772323b8c41b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.835287 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 35101a6997e24040994661756d9649d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.834869 886 tablet_service.cc:1505] Processing CreateTablet for tablet 144dee1a6d7344f2b79da6e79e67599e (DEFAULT_TABLE table=default.Table_c16ccd85e43049d69fa14c92e073791c [id=8cd62da3d57c4be4b5f10ace49d2e148]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.835837 895 tablet_service.cc:1552] Processing DeleteTablet for tablet dc442726ec0c456290ecf2892d0df957 with delete_type TABLET_DATA_DELETED (replaced by tablet 6bd6856b800247de896f16eee66f2827) from {username='slave'} at 127.0.0.1:43856
W20251214 20:52:10.836026 712 leader_election.cc:343] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.836488 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2ee5cc95304fc1a80ed4d31838ac38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.838593 895 tablet_service.cc:1505] Processing CreateTablet for tablet 25974e5db555447baf5799775d15cd14 (DEFAULT_TABLE table=default.Table_209aeff6e9694fc6a42598018aba1dc1 [id=7dbd8a94ed0b497d8eba92091819fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.838853 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63497e77baeb4d328be2421b5732a28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.839498 755 tablet_service.cc:1505] Processing CreateTablet for tablet b1df0dad2fb040a5a12e0401e00c87db (DEFAULT_TABLE table=default.Table_26a97fa4596e4f769036f616c0d377df [id=ed335fb680bc441184e1682200ee68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.839884 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1df0dad2fb040a5a12e0401e00c87db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.840440 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_01dcfe83d7854b4e9b8d6f62a557a0fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.840976 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01dcfe83d7854b4e9b8d6f62a557a0fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.841513 742 tablet_service.cc:1505] Processing CreateTablet for tablet ba50e32f7e724868bb891045a939e362 (DEFAULT_TABLE table=default.Table_348a96aca6644d08b3074727ff3df5a7 [id=c3e5006d02fc4b69987feb30ba66d069]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.841857 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba50e32f7e724868bb891045a939e362. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.842974 1217 raft_consensus.cc:493] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:10.843222 1217 raft_consensus.cc:515] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.843961 1217 leader_election.cc:290] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:10.845348 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a3c01a9acc334c4cba07c4646084e233" } new_table_name: "default.Table_b01c45a1fe8f4a71858fcb2127177471" modify_external_catalogs: true
I20251214 20:52:10.847208 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d553ca568da242899d768f9848baba3d" } modify_external_catalogs: true
I20251214 20:52:10.847406 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d553ca568da242899d768f9848baba3d" } modify_external_catalogs: true
I20251214 20:52:10.853606 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_6061b180a35048c4a55552330aef8b00" } new_table_name: "default.Table_1437581b4e464656a47dc3ab99914c22" modify_external_catalogs: true
I20251214 20:52:10.855582 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:10.858006 755 tablet_service.cc:1505] Processing CreateTablet for tablet 6bd6856b800247de896f16eee66f2827 (DEFAULT_TABLE table=default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.858568 1202 tablet_bootstrap.cc:492] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:10.858549 751 tablet_service.cc:1505] Processing CreateTablet for tablet c25832da5c1644329e18653000f3330e (DEFAULT_TABLE table=default.Table_017c57bf3637493fb9ccb5976322bfe7 [id=5427e484a5a94314a13be3f387c06d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.858718 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd6856b800247de896f16eee66f2827. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.859016 749 tablet_service.cc:1505] Processing CreateTablet for tablet 3fa0317e999b483788e5c5652e6e83de (DEFAULT_TABLE table=default.Table_a3c01a9acc334c4cba07c4646084e233 [id=ef22180b36a241fea13d0e622c02b50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.859772 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa0317e999b483788e5c5652e6e83de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.860146 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba2ed363bbab4f1f865b183297c98daa" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:10.861408 712 leader_election.cc:343] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:10.861712 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144dee1a6d7344f2b79da6e79e67599e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.862985 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25974e5db555447baf5799775d15cd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.863245 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25832da5c1644329e18653000f3330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.869256 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 6cc54be0bc6d4f259967fbe0a5023f82 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.873580 756 tablet_service.cc:1505] Processing CreateTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 (DEFAULT_TABLE table=default.Table_a2a64eab2d9a493da6fee1051eadd37d [id=d929401b9c4c4b4ba21353fb8a8ecc3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.874132 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2842f1dc0c374d24946f90dac1cf8aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.875852 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_76a2939ccdea4fdeaa1da24713bc3ca7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.876529 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76a2939ccdea4fdeaa1da24713bc3ca7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.877183 742 tablet_service.cc:1505] Processing CreateTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d (DEFAULT_TABLE table=default.Table_ca875b1006554ba6845c0cf23671a8da [id=8c192689a28f4314bb5608c08ea21a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.877671 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d768c45bc02b4d108798efbf3f6aaf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.877725 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 8e2ee5cc95304fc1a80ed4d31838ac38 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:10.879482 747 tablet_service.cc:1505] Processing CreateTablet for tablet a142157acbcd44adb6099b4428103ee5 (DEFAULT_TABLE table=default.Table_efa9fa90866441c8a4d8a14d14bb4bdd [id=0a848dd5f6474712b2eba70add3743dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.879899 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a142157acbcd44adb6099b4428103ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.883163 1202 tablet_bootstrap.cc:654] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:10.886024 1206 tablet_bootstrap.cc:492] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:10.887180 1206 ts_tablet_manager.cc:1403] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.002s
I20251214 20:52:10.887954 749 tablet_service.cc:1505] Processing CreateTablet for tablet f0d661a1bb034053a4634ff537eafc7d (DEFAULT_TABLE table=default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.888247 1206 raft_consensus.cc:359] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.888597 1206 raft_consensus.cc:385] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:10.888723 1206 raft_consensus.cc:740] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:10.889109 751 tablet_service.cc:1505] Processing CreateTablet for tablet a0279992d1f2419ab1caae116a0391b2 (DEFAULT_TABLE table=default.Table_4e46507057c94aa7a347491bf2e482ff [id=3906806ef8294342a79b9face8753212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.889379 754 tablet_service.cc:1505] Processing CreateTablet for tablet 649f5bd67cbb4f61b7066269f0bf9266 (DEFAULT_TABLE table=default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.889953 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 649f5bd67cbb4f61b7066269f0bf9266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.890067 1206 consensus_queue.cc:260] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:10.892927 885 tablet_service.cc:1505] Processing CreateTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc (DEFAULT_TABLE table=default.Table_1d198a3975c24b84aaeb82c266c746b8 [id=bc62e2586be140efbc2bfb5a3b5baaa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.893759 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8faa7fd7315f485c82aee38bcf4cd8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.894161 883 tablet_service.cc:1505] Processing CreateTablet for tablet a142157acbcd44adb6099b4428103ee5 (DEFAULT_TABLE table=default.Table_efa9fa90866441c8a4d8a14d14bb4bdd [id=0a848dd5f6474712b2eba70add3743dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.895361 882 tablet_service.cc:1505] Processing CreateTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d (DEFAULT_TABLE table=default.Table_ca875b1006554ba6845c0cf23671a8da [id=8c192689a28f4314bb5608c08ea21a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.896168 881 tablet_service.cc:1505] Processing CreateTablet for tablet c25832da5c1644329e18653000f3330e (DEFAULT_TABLE table=default.Table_017c57bf3637493fb9ccb5976322bfe7 [id=5427e484a5a94314a13be3f387c06d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.896314 880 tablet_service.cc:1505] Processing CreateTablet for tablet 3fa0317e999b483788e5c5652e6e83de (DEFAULT_TABLE table=default.Table_a3c01a9acc334c4cba07c4646084e233 [id=ef22180b36a241fea13d0e622c02b50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.897189 879 tablet_service.cc:1505] Processing CreateTablet for tablet f0d661a1bb034053a4634ff537eafc7d (DEFAULT_TABLE table=default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.897610 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa0317e999b483788e5c5652e6e83de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.897881 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d661a1bb034053a4634ff537eafc7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.899266 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d768c45bc02b4d108798efbf3f6aaf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.899732 888 tablet_service.cc:1505] Processing CreateTablet for tablet a0279992d1f2419ab1caae116a0391b2 (DEFAULT_TABLE table=default.Table_4e46507057c94aa7a347491bf2e482ff [id=3906806ef8294342a79b9face8753212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.893918 884 tablet_service.cc:1505] Processing CreateTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 (DEFAULT_TABLE table=default.Table_a2a64eab2d9a493da6fee1051eadd37d [id=d929401b9c4c4b4ba21353fb8a8ecc3e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:10.901427 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 6381cbf05a8c4f84b71f229497fd28ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6381cbf05a8c4f84b71f229497fd28ef already in progress: creating tablet
W20251214 20:52:10.901665 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 0a0ca65d369a45958c6dac539964114b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a0ca65d369a45958c6dac539964114b already in progress: creating tablet
W20251214 20:52:10.901835 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 5ba2d34a6d2645bd97fe85fed952692b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ba2d34a6d2645bd97fe85fed952692b already in progress: creating tablet
W20251214 20:52:10.902001 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet dc442726ec0c456290ecf2892d0df957 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc442726ec0c456290ecf2892d0df957 already in progress: creating tablet
I20251214 20:52:10.904035 878 tablet_service.cc:1505] Processing CreateTablet for tablet 649f5bd67cbb4f61b7066269f0bf9266 (DEFAULT_TABLE table=default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.904259 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0279992d1f2419ab1caae116a0391b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.904524 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a142157acbcd44adb6099b4428103ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.906725 877 tablet_service.cc:1552] Processing DeleteTablet for tablet 649f5bd67cbb4f61b7066269f0bf9266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.907871 887 tablet_service.cc:1505] Processing CreateTablet for tablet b1df0dad2fb040a5a12e0401e00c87db (DEFAULT_TABLE table=default.Table_26a97fa4596e4f769036f616c0d377df [id=ed335fb680bc441184e1682200ee68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.888401 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d661a1bb034053a4634ff537eafc7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.909852 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 649f5bd67cbb4f61b7066269f0bf9266. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:10.910713 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f2b66900d6ea42aa849c24acbbe23c40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2b66900d6ea42aa849c24acbbe23c40 already in progress: creating tablet
I20251214 20:52:10.911669 1207 ts_tablet_manager.cc:1916] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:10.912169 748 tablet_service.cc:1552] Processing DeleteTablet for tablet a05f1ec28fb24508ae937c70aaaefffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.916759 526 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 6b4f582d8165439d86d9228c146dc41a
I20251214 20:52:10.917088 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_9f6bb794724346069627cd2f14a5cd1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.917793 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f6bb794724346069627cd2f14a5cd1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:10.918354 523 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fa0317e999b483788e5c5652e6e83de consensus state has no leader
I20251214 20:52:10.918740 521 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 63497e77baeb4d328be2421b5732a28a
W20251214 20:52:10.919479 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12b5f4326aa0413eb303c132ba21decd consensus state has no leader
I20251214 20:52:10.925159 517 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet ff9da2edb6ba4bf28448e7e6ce48443d
I20251214 20:52:10.927277 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25832da5c1644329e18653000f3330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.928177 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0279992d1f2419ab1caae116a0391b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.926749 892 tablet_service.cc:1505] Processing CreateTablet for tablet ba50e32f7e724868bb891045a939e362 (DEFAULT_TABLE table=default.Table_348a96aca6644d08b3074727ff3df5a7 [id=c3e5006d02fc4b69987feb30ba66d069]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.930078 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_fa0e830600ba474bbbd6a68b9593d5f2" } modify_external_catalogs: true
I20251214 20:52:10.930287 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_fa0e830600ba474bbbd6a68b9593d5f2" } modify_external_catalogs: true
W20251214 20:52:10.931159 512 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 649f5bd67cbb4f61b7066269f0bf9266 because the tablet was not found. No further retry: Not found: Tablet not found: 649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:10.934645 1206 ts_tablet_manager.cc:1434] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.045s user 0.000s sys 0.003s
I20251214 20:52:10.939080 893 tablet_service.cc:1505] Processing CreateTablet for tablet 6bd6856b800247de896f16eee66f2827 (DEFAULT_TABLE table=default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.941112 877 tablet_service.cc:1552] Processing DeleteTablet for tablet 63497e77baeb4d328be2421b5732a28a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.942467 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 649f5bd67cbb4f61b7066269f0bf9266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.943194 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 63497e77baeb4d328be2421b5732a28a (DEFAULT_TABLE table=default.Table_d553ca568da242899d768f9848baba3d [id=376e4eabd04d4a53ab2772323b8c41b7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:10.945780 512 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 63497e77baeb4d328be2421b5732a28a because the tablet was not found. No further retry: Not found: Tablet not found: 63497e77baeb4d328be2421b5732a28a
I20251214 20:52:10.948102 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1df0dad2fb040a5a12e0401e00c87db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.950857 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd6856b800247de896f16eee66f2827. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.951800 1202 tablet_bootstrap.cc:492] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:10.951872 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba50e32f7e724868bb891045a939e362. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.952003 1202 ts_tablet_manager.cc:1403] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.127s user 0.002s sys 0.002s
I20251214 20:52:10.956625 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2842f1dc0c374d24946f90dac1cf8aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.956992 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 6cc54be0bc6d4f259967fbe0a5023f82
I20251214 20:52:10.957525 524 catalog_manager.cc:6724] ReplaceTablet: tablet 6cc54be0bc6d4f259967fbe0a5023f82 deleted and replaced by tablet 4a625fcb95324466a12ab118bdb859f1
I20251214 20:52:10.958119 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 8e2ee5cc95304fc1a80ed4d31838ac38
I20251214 20:52:10.958431 520 catalog_manager.cc:6724] ReplaceTablet: tablet 8e2ee5cc95304fc1a80ed4d31838ac38 deleted and replaced by tablet 9cfd9b2f9756433e9e52643417c4cd45
I20251214 20:52:10.961899 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 023299180d074893847e84dcd9812751 (DEFAULT_TABLE table=default.Table_274bf0a7631e4c3c8144694db10a5de9 [id=c0729ac770394d7b87ef36e53f85c151]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.966420 877 tablet_service.cc:1552] Processing DeleteTablet for tablet 6cc54be0bc6d4f259967fbe0a5023f82 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a625fcb95324466a12ab118bdb859f1) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.972677 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cfd9b2f9756433e9e52643417c4cd45) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.973186 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_997191dc162f47a4b9e3b05289f6cb48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:10.973835 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_997191dc162f47a4b9e3b05289f6cb48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:10.977793 1206 tablet_bootstrap.cc:492] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:10.981508 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 6cc54be0bc6d4f259967fbe0a5023f82 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a625fcb95324466a12ab118bdb859f1) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.982990 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cfd9b2f9756433e9e52643417c4cd45) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.983546 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 63497e77baeb4d328be2421b5732a28a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.984928 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b4f582d8165439d86d9228c146dc41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:10.985909 748 tablet_service.cc:1505] Processing CreateTablet for tablet b652a8a80246443491551e76e68d2a8b (DEFAULT_TABLE table=default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:10.988430 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b652a8a80246443491551e76e68d2a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:10.989035 525 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 47a67b1b9dc94e498cad91a72e655a75
I20251214 20:52:10.990756 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b4f582d8165439d86d9228c146dc41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:10.991631 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_348a96aca6644d08b3074727ff3df5a7" } new_table_name: "default.Table_8e3f7e1435ea493fa4729ca0d9e23573" modify_external_catalogs: true
I20251214 20:52:10.999877 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 47a67b1b9dc94e498cad91a72e655a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.000046 876 tablet_service.cc:1505] Processing CreateTablet for tablet b652a8a80246443491551e76e68d2a8b (DEFAULT_TABLE table=default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.002734 886 tablet_service.cc:1505] Processing CreateTablet for tablet f3a2143b5508445492b970d2588cf1cf (DEFAULT_TABLE table=default.Table_d653f9c533784976ac955f90dc39eb56 [id=295a34df92b44b738e3aa3d1d8195673]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.002851 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 47a67b1b9dc94e498cad91a72e655a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:11.003202 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b652a8a80246443491551e76e68d2a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.003259 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 7bfb575249b8403991d21c6fc96eefeb from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:11.005055 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_209aeff6e9694fc6a42598018aba1dc1" } new_table_name: "default.Table_e744815cb34441df8b96e5ab9527681e" modify_external_catalogs: true
I20251214 20:52:11.005364 1208 tablet_replica.cc:333] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:11.008102 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_bff6195ce6594ee9b4a55eb6058dcb63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:11.008342 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_8076bc8b31ba411faf089f314e6ed2ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:11.008774 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8076bc8b31ba411faf089f314e6ed2ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.010003 1202 raft_consensus.cc:359] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.010492 1202 raft_consensus.cc:385] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.010665 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3900abc2f8dc49ceb70dc23666566d3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:11.011161 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3900abc2f8dc49ceb70dc23666566d3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.012455 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_69c0719fe94a4c7594516a739f6a40ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:11.012565 1208 raft_consensus.cc:2243] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:11.012887 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69c0719fe94a4c7594516a739f6a40ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.013933 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_26a97fa4596e4f769036f616c0d377df" } new_table_name: "default.Table_d78784658d7d47e1b55c72607d6169b0" modify_external_catalogs: true
I20251214 20:52:11.015081 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_cae5b2188af54d068db115a0745ee866"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:11.015679 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cae5b2188af54d068db115a0745ee866 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:11.008776 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bff6195ce6594ee9b4a55eb6058dcb63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.021322 1207 ts_tablet_manager.cc:1929] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:11.022116 1211 raft_consensus.cc:493] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.023576 1207 log.cc:1199] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/6381cbf05a8c4f84b71f229497fd28ef
I20251214 20:52:11.024454 1027 tablet_service.cc:1505] Processing CreateTablet for tablet b1df0dad2fb040a5a12e0401e00c87db (DEFAULT_TABLE table=default.Table_26a97fa4596e4f769036f616c0d377df [id=ed335fb680bc441184e1682200ee68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.025298 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 6b4f582d8165439d86d9228c146dc41a (DEFAULT_TABLE table=default.Table_22df18c031ab4778ae90eb3580e9b8b7 [id=3f105cc6dd6944ceb9783a1ebe923d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.025667 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 (DEFAULT_TABLE table=default.Table_a2a64eab2d9a493da6fee1051eadd37d [id=d929401b9c4c4b4ba21353fb8a8ecc3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.026144 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63497e77baeb4d328be2421b5732a28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.026522 1206 tablet_bootstrap.cc:654] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.028393 1202 raft_consensus.cc:740] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.029017 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a2143b5508445492b970d2588cf1cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:11.030938 513 catalog_manager.cc:5058] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): alter failed for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (table default.Table_efa9fa90866441c8a4d8a14d14bb4bdd [id=0a848dd5f6474712b2eba70add3743dc]),no further retry: Not found: Tablet not found: 2a0eb9ab79d9428997c6f66790d6b1a3
I20251214 20:52:11.031435 1029 tablet_service.cc:1505] Processing CreateTablet for tablet c25832da5c1644329e18653000f3330e (DEFAULT_TABLE table=default.Table_017c57bf3637493fb9ccb5976322bfe7 [id=5427e484a5a94314a13be3f387c06d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.031489 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a0ca65d369a45958c6dac539964114b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:11.032572 1202 consensus_queue.cc:260] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.033047 1211 raft_consensus.cc:515] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.033188 1208 raft_consensus.cc:2272] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.034137 1203 tablet_bootstrap.cc:492] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:11.035066 1034 tablet_service.cc:1505] Processing CreateTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d (DEFAULT_TABLE table=default.Table_ca875b1006554ba6845c0cf23671a8da [id=8c192689a28f4314bb5608c08ea21a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.035472 1026 tablet_service.cc:1505] Processing CreateTablet for tablet a142157acbcd44adb6099b4428103ee5 (DEFAULT_TABLE table=default.Table_efa9fa90866441c8a4d8a14d14bb4bdd [id=0a848dd5f6474712b2eba70add3743dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.035535 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 6bd6856b800247de896f16eee66f2827 (DEFAULT_TABLE table=default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.035702 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 3fa0317e999b483788e5c5652e6e83de (DEFAULT_TABLE table=default.Table_a3c01a9acc334c4cba07c4646084e233 [id=ef22180b36a241fea13d0e622c02b50a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.036149 1023 tablet_service.cc:1505] Processing CreateTablet for tablet a0279992d1f2419ab1caae116a0391b2 (DEFAULT_TABLE table=default.Table_4e46507057c94aa7a347491bf2e482ff [id=3906806ef8294342a79b9face8753212]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.036571 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1df0dad2fb040a5a12e0401e00c87db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.036870 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a142157acbcd44adb6099b4428103ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.036918 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 649f5bd67cbb4f61b7066269f0bf9266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.037182 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 6cc54be0bc6d4f259967fbe0a5023f82 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a625fcb95324466a12ab118bdb859f1) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.037389 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cfd9b2f9756433e9e52643417c4cd45) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.037940 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023299180d074893847e84dcd9812751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.038189 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d768c45bc02b4d108798efbf3f6aaf7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.039211 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b4f582d8165439d86d9228c146dc41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.040550 1017 tablet_service.cc:1505] Processing CreateTablet for tablet b652a8a80246443491551e76e68d2a8b (DEFAULT_TABLE table=default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.040959 1056 raft_consensus.cc:2468] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:11.039224 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25832da5c1644329e18653000f3330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.042023 713 leader_election.cc:304] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:11.042541 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd6856b800247de896f16eee66f2827. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.042590 1217 raft_consensus.cc:2804] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:11.042675 1217 raft_consensus.cc:493] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:11.042775 1217 raft_consensus.cc:3060] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.043071 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa0317e999b483788e5c5652e6e83de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.043973 1208 ts_tablet_manager.cc:1916] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.045701 1217 raft_consensus.cc:515] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.046766 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2842f1dc0c374d24946f90dac1cf8aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.046947 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a0ca65d369a45958c6dac539964114b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
W20251214 20:52:11.047559 712 leader_election.cc:343] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:11.048368 1217 leader_election.cc:290] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.049418 1202 ts_tablet_manager.cc:1434] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.083s user 0.003s sys 0.000s
I20251214 20:52:11.050045 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a0ca65d369a45958c6dac539964114b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:11.050284 1056 raft_consensus.cc:3060] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.052876 1056 raft_consensus.cc:2468] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:11.053395 713 leader_election.cc:304] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:11.053848 1217 raft_consensus.cc:2804] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:11.054013 1217 raft_consensus.cc:697] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:11.054316 1217 consensus_queue.cc:237] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.058086 1202 tablet_bootstrap.cc:492] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.060325 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 63497e77baeb4d328be2421b5732a28a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.063261 1203 ts_tablet_manager.cc:1403] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.346s user 0.021s sys 0.028s
I20251214 20:52:11.065128 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a0ca65d369a45958c6dac539964114b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:11.066490 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0279992d1f2419ab1caae116a0391b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.067183 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a0ca65d369a45958c6dac539964114b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:11.067898 991 leader_election.cc:343] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:11.069901 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a0ca65d369a45958c6dac539964114b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.070429 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 47a67b1b9dc94e498cad91a72e655a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.071084 989 leader_election.cc:304] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:11.071540 1211 leader_election.cc:290] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:11.072331 1205 ts_tablet_manager.cc:1929] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:11.072795 1211 raft_consensus.cc:2749] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:11.076424 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 649f5bd67cbb4f61b7066269f0bf9266 (DEFAULT_TABLE table=default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.077173 882 tablet_service.cc:1505] Processing CreateTablet for tablet 40a9a6192c744282ab5672cbe15fd299 (DEFAULT_TABLE table=default.Table_847906e708bd41e9b9507efc8f52347f [id=4afcd94192854e049c21f8e6852cd56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.077282 755 tablet_service.cc:1505] Processing CreateTablet for tablet 40a9a6192c744282ab5672cbe15fd299 (DEFAULT_TABLE table=default.Table_847906e708bd41e9b9507efc8f52347f [id=4afcd94192854e049c21f8e6852cd56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.077426 1021 tablet_service.cc:1505] Processing CreateTablet for tablet f3a2143b5508445492b970d2588cf1cf (DEFAULT_TABLE table=default.Table_d653f9c533784976ac955f90dc39eb56 [id=295a34df92b44b738e3aa3d1d8195673]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.077752 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a9a6192c744282ab5672cbe15fd299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.077795 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a9a6192c744282ab5672cbe15fd299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.087244 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_5242f3b282284ab7aff8525106732d53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:11.090656 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5242f3b282284ab7aff8525106732d53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.091576 889 tablet_service.cc:1505] Processing CreateTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 (DEFAULT_TABLE table=default.Table_cb89b84fb9ec4a0d89a3fd65d538725e [id=2018af6df0b448bd8996691cb5a1f62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.092061 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd88d1e95ee04ef596863e1ae3687de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.094807 1207 ts_tablet_manager.cc:1950] T 6381cbf05a8c4f84b71f229497fd28ef P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:11.095825 745 tablet_service.cc:1505] Processing CreateTablet for tablet f3a2143b5508445492b970d2588cf1cf (DEFAULT_TABLE table=default.Table_d653f9c533784976ac955f90dc39eb56 [id=295a34df92b44b738e3aa3d1d8195673]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.096654 880 tablet_service.cc:1505] Processing CreateTablet for tablet 8d74d3abc45a41969289c0a0892bfa04 (DEFAULT_TABLE table=default.Table_18dc9de393304efcb3ec4eb6be858624 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.097113 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d74d3abc45a41969289c0a0892bfa04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.098004 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 6381cbf05a8c4f84b71f229497fd28ef (table default.Table_035a23623ce242298ab89c51024d7a09 [id=cd7616f6ff3244ec8628da5102e51570]) successfully deleted
I20251214 20:52:11.100908 1207 tablet_replica.cc:333] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:11.101428 756 tablet_service.cc:1505] Processing CreateTablet for tablet 8d74d3abc45a41969289c0a0892bfa04 (DEFAULT_TABLE table=default.Table_18dc9de393304efcb3ec4eb6be858624 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.103267 878 tablet_service.cc:1552] Processing DeleteTablet for tablet cefcb15849d1431fb50790343f763ba1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.104249 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a2143b5508445492b970d2588cf1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.104938 1202 tablet_bootstrap.cc:654] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.106706 1207 raft_consensus.cc:2243] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:11.108290 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d74d3abc45a41969289c0a0892bfa04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.110122 1207 raft_consensus.cc:2272] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.115315 1207 ts_tablet_manager.cc:1916] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.118350 1274 raft_consensus.cc:493] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.118541 1274 raft_consensus.cc:515] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.121953 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a05f1ec28fb24508ae937c70aaaefffe" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:11.123637 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:35099: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111),
I20251214 20:52:11.123831 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251214 20:52:11.126286 888 tablet_service.cc:1505] Processing CreateTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 (DEFAULT_TABLE table=default.Table_03651b0344ca49b688101a958c2090e4 [id=b16533ff3115482d9357bd88bc1406cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.126889 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472e4a5990e64d02b1d52dc6e86d7190. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.133298 1274 leader_election.cc:290] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:11.134007 712 leader_election.cc:343] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:11.137127 1202 tablet_bootstrap.cc:492] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.137483 1202 ts_tablet_manager.cc:1403] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.000s
I20251214 20:52:11.138444 1202 raft_consensus.cc:359] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.138710 1202 raft_consensus.cc:385] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.138787 1202 raft_consensus.cc:740] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.138666 742 tablet_service.cc:1505] Processing CreateTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 (DEFAULT_TABLE table=default.Table_cb89b84fb9ec4a0d89a3fd65d538725e [id=2018af6df0b448bd8996691cb5a1f62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.139523 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd88d1e95ee04ef596863e1ae3687de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.140024 1202 consensus_queue.cc:260] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.149142 749 tablet_service.cc:1505] Processing CreateTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 (DEFAULT_TABLE table=default.Table_03651b0344ca49b688101a958c2090e4 [id=b16533ff3115482d9357bd88bc1406cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.149670 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472e4a5990e64d02b1d52dc6e86d7190. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.151552 1202 ts_tablet_manager.cc:1434] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20251214 20:52:11.154861 1202 tablet_bootstrap.cc:492] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.156462 1208 ts_tablet_manager.cc:1929] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:11.156615 1208 log.cc:1199] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/6cc54be0bc6d4f259967fbe0a5023f82
I20251214 20:52:11.158888 1208 ts_tablet_manager.cc:1950] T 6cc54be0bc6d4f259967fbe0a5023f82 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:11.170084 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]) successfully deleted
I20251214 20:52:11.158453 1202 tablet_bootstrap.cc:654] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:11.171825 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e2ee5cc95304fc1a80ed4d31838ac38 already in progress: creating tablet
W20251214 20:52:11.172483 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 6b4f582d8165439d86d9228c146dc41a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b4f582d8165439d86d9228c146dc41a already in progress: creating tablet
W20251214 20:52:11.173199 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 47a67b1b9dc94e498cad91a72e655a75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47a67b1b9dc94e498cad91a72e655a75 already in progress: creating tablet
I20251214 20:52:11.173506 1208 tablet_replica.cc:333] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:11.173730 1208 raft_consensus.cc:2243] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:11.173921 1208 raft_consensus.cc:2272] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.178650 1202 tablet_bootstrap.cc:492] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.178824 1202 ts_tablet_manager.cc:1403] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.002s
I20251214 20:52:11.179654 1202 raft_consensus.cc:359] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.179857 1202 raft_consensus.cc:385] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.179952 1202 raft_consensus.cc:740] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.180217 1202 consensus_queue.cc:260] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.182310 1202 ts_tablet_manager.cc:1434] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:11.183506 1202 tablet_bootstrap.cc:492] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.185357 1202 tablet_bootstrap.cc:654] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.188283 1202 tablet_bootstrap.cc:492] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.188459 1202 ts_tablet_manager.cc:1403] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251214 20:52:11.190503 1202 raft_consensus.cc:359] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.190707 1202 raft_consensus.cc:385] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.190775 1202 raft_consensus.cc:740] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.191068 1202 consensus_queue.cc:260] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:11.191951 520 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba50e32f7e724868bb891045a939e362 consensus state has no leader
I20251214 20:52:11.194312 1202 ts_tablet_manager.cc:1434] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251214 20:52:11.195591 1202 tablet_bootstrap.cc:492] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.195654 520 master_service.cc:954] ReplaceTablet: received request to replace tablet ba2ed363bbab4f1f865b183297c98daa from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:11.196301 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_e3603e6d524c40dea2a1a88b79d63a8e" } modify_external_catalogs: true
I20251214 20:52:11.196537 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_e3603e6d524c40dea2a1a88b79d63a8e" } modify_external_catalogs: true
I20251214 20:52:11.199246 1202 tablet_bootstrap.cc:654] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.207472 1202 tablet_bootstrap.cc:492] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.207660 1202 ts_tablet_manager.cc:1403] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20251214 20:52:11.208568 1202 raft_consensus.cc:359] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.208773 1202 raft_consensus.cc:385] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.208870 1202 raft_consensus.cc:740] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.209139 1202 consensus_queue.cc:260] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.214128 1202 ts_tablet_manager.cc:1434] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20251214 20:52:11.216302 518 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 25974e5db555447baf5799775d15cd14 consensus state has no leader
I20251214 20:52:11.218336 1202 tablet_bootstrap.cc:492] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.220937 888 tablet_service.cc:1552] Processing DeleteTablet for tablet ac19d32a31e7405b9f16566215d6b2cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.221208 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 35101a6997e24040994661756d9649d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.221411 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a0eb9ab79d9428997c6f66790d6b1a3 with delete_type TABLET_DATA_DELETED (replaced by tablet a142157acbcd44adb6099b4428103ee5) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.221581 1202 tablet_bootstrap.cc:654] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.223573 1207 ts_tablet_manager.cc:1929] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:11.223789 1207 log.cc:1199] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ac19d32a31e7405b9f16566215d6b2cd
I20251214 20:52:11.227056 1208 ts_tablet_manager.cc:1916] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.232461 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_9f6bb794724346069627cd2f14a5cd1b" } modify_external_catalogs: true
I20251214 20:52:11.232717 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_9f6bb794724346069627cd2f14a5cd1b" } modify_external_catalogs: true
I20251214 20:52:11.241729 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d653f9c533784976ac955f90dc39eb56" } modify_external_catalogs: true
I20251214 20:52:11.242401 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d653f9c533784976ac955f90dc39eb56" } modify_external_catalogs: true
I20251214 20:52:11.247687 1207 ts_tablet_manager.cc:1950] T ac19d32a31e7405b9f16566215d6b2cd P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:11.248307 1207 tablet_replica.cc:333] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:11.248421 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ac19d32a31e7405b9f16566215d6b2cd (table default.Table_8090a0016b2248f6b8fa395e4f676734 [id=ac9c660a341b4fe6abedd5343196a30a]) successfully deleted
I20251214 20:52:11.230032 1202 tablet_bootstrap.cc:492] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.249231 1202 ts_tablet_manager.cc:1403] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.007s
I20251214 20:52:11.250114 1202 raft_consensus.cc:359] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.250336 1202 raft_consensus.cc:385] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.250432 1202 raft_consensus.cc:740] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.250706 1202 consensus_queue.cc:260] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.241933 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a0ca65d369a45958c6dac539964114b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.255116 1202 ts_tablet_manager.cc:1434] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:11.255002 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 40a9a6192c744282ab5672cbe15fd299 (DEFAULT_TABLE table=default.Table_847906e708bd41e9b9507efc8f52347f [id=4afcd94192854e049c21f8e6852cd56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.255486 1202 tablet_bootstrap.cc:492] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.257087 1202 tablet_bootstrap.cc:654] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.259012 1205 log.cc:1199] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/23ecf83d935c49f9b304a3355b0cc55d
I20251214 20:52:11.261622 1031 tablet_service.cc:1505] Processing CreateTablet for tablet f0d661a1bb034053a4634ff537eafc7d (DEFAULT_TABLE table=default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.263615 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b652a8a80246443491551e76e68d2a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.067905 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba2ed363bbab4f1f865b183297c98daa" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:11.274016 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 204377us
I20251214 20:52:11.278929 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a05f1ec28fb24508ae937c70aaaefffe" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:11.279858 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2842f1dc0c374d24946f90dac1cf8aa5 (table default.Table_a2a64eab2d9a493da6fee1051eadd37d [id=d929401b9c4c4b4ba21353fb8a8ecc3e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2842f1dc0c374d24946f90dac1cf8aa5 consensus state has no leader
I20251214 20:52:11.279891 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 8d74d3abc45a41969289c0a0892bfa04 (DEFAULT_TABLE table=default.Table_18dc9de393304efcb3ec4eb6be858624 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:11.281793 713 leader_election.cc:343] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:11.281983 713 leader_election.cc:304] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:11.282528 713 leader_election.cc:343] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:11.282639 713 leader_election.cc:304] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:11.283282 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 649f5bd67cbb4f61b7066269f0bf9266. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.284682 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b4f582d8165439d86d9228c146dc41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.287276 1205 ts_tablet_manager.cc:1950] T 23ecf83d935c49f9b304a3355b0cc55d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:11.297569 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 23ecf83d935c49f9b304a3355b0cc55d (table default.Table_b934001f6c8e4e7b86b4dbcbca5d066d [id=6d4f571defdf455c8575302706c2a2b0]) successfully deleted
I20251214 20:52:11.299467 1274 raft_consensus.cc:2749] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:11.299816 1274 raft_consensus.cc:2749] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:11.308862 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f2b66900d6ea42aa849c24acbbe23c40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2b66900d6ea42aa849c24acbbe23c40 already in progress: creating tablet
I20251214 20:52:11.307756 1203 raft_consensus.cc:359] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.315223 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 (DEFAULT_TABLE table=default.Table_03651b0344ca49b688101a958c2090e4 [id=b16533ff3115482d9357bd88bc1406cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.315511 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a2143b5508445492b970d2588cf1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.316363 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472e4a5990e64d02b1d52dc6e86d7190. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.316602 1213 raft_consensus.cc:493] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.316828 1213 raft_consensus.cc:515] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.317579 1213 leader_election.cc:290] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.318744 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba2ed363bbab4f1f865b183297c98daa" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:11.319046 777 raft_consensus.cc:2468] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:11.319182 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d74d3abc45a41969289c0a0892bfa04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.319880 848 leader_election.cc:304] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:11.320246 1213 raft_consensus.cc:2804] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:11.320390 1213 raft_consensus.cc:493] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:11.320495 1213 raft_consensus.cc:3060] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.318619 1203 raft_consensus.cc:385] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.322269 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a9a6192c744282ab5672cbe15fd299. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.324549 1213 raft_consensus.cc:515] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.328125 1213 leader_election.cc:290] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.328408 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba2ed363bbab4f1f865b183297c98daa" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:11.328722 777 raft_consensus.cc:3060] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.332087 777 raft_consensus.cc:2468] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:11.332973 848 leader_election.cc:304] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:11.346884 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba2ed363bbab4f1f865b183297c98daa" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:11.348754 1213 raft_consensus.cc:2804] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:11.355835 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d661a1bb034053a4634ff537eafc7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.357306 1213 raft_consensus.cc:697] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:11.357892 1213 consensus_queue.cc:237] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.379187 1030 tablet_service.cc:1505] Processing CreateTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 (DEFAULT_TABLE table=default.Table_cb89b84fb9ec4a0d89a3fd65d538725e [id=2018af6df0b448bd8996691cb5a1f62c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.384661 1203 raft_consensus.cc:740] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:11.387181 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 104563us
I20251214 20:52:11.388878 777 raft_consensus.cc:3060] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.391620 1203 consensus_queue.cc:260] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.392738 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba2ed363bbab4f1f865b183297c98daa" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:11.396433 1205 tablet_replica.cc:333] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:11.398353 1206 tablet_bootstrap.cc:492] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:11.398557 1206 ts_tablet_manager.cc:1403] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.450s user 0.010s sys 0.007s
I20251214 20:52:11.399576 1206 raft_consensus.cc:359] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.399763 1206 raft_consensus.cc:385] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.399838 1206 raft_consensus.cc:740] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.400045 1206 consensus_queue.cc:260] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.401175 1206 ts_tablet_manager.cc:1434] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:11.405339 1213 raft_consensus.cc:493] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.405576 1213 raft_consensus.cc:515] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.406267 1213 leader_election.cc:290] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.407253 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd88d1e95ee04ef596863e1ae3687de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.410679 1205 raft_consensus.cc:2243] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:11.411412 1056 raft_consensus.cc:2468] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:11.419404 1203 ts_tablet_manager.cc:1434] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.329s user 0.002s sys 0.002s
I20251214 20:52:11.419490 1202 tablet_bootstrap.cc:492] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.419661 1202 ts_tablet_manager.cc:1403] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.164s user 0.060s sys 0.089s
I20251214 20:52:11.420459 1202 raft_consensus.cc:359] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.420634 1202 raft_consensus.cc:385] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.420732 1202 raft_consensus.cc:740] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.421473 1213 raft_consensus.cc:493] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.421612 1213 raft_consensus.cc:515] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.422211 1213 leader_election.cc:290] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.424135 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_3c8ed47b57e546f59c9d863f953a17ec" } new_table_name: "default.Table_2162a5b765254842bb0b7e4ba7f66136" modify_external_catalogs: true
I20251214 20:52:11.424559 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_03651b0344ca49b688101a958c2090e4" } new_table_name: "default.Table_2acb51a56a60469e941100c99494a99b" modify_external_catalogs: true
I20251214 20:52:11.425992 525 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 7bfb575249b8403991d21c6fc96eefeb
I20251214 20:52:11.426436 525 catalog_manager.cc:6724] ReplaceTablet: tablet 7bfb575249b8403991d21c6fc96eefeb deleted and replaced by tablet 2b92b6d5dde2465684d19aee44a83160
I20251214 20:52:11.426759 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_2325a2c5e97f45ebac4c943380bb3525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:11.426972 1202 consensus_queue.cc:260] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:11.427414 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2325a2c5e97f45ebac4c943380bb3525 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:11.428814 524 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1df0dad2fb040a5a12e0401e00c87db consensus state has no leader
I20251214 20:52:11.429157 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a2a64eab2d9a493da6fee1051eadd37d" } modify_external_catalogs: true
I20251214 20:52:11.429329 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a2a64eab2d9a493da6fee1051eadd37d" } modify_external_catalogs: true
I20251214 20:52:11.429963 1208 ts_tablet_manager.cc:1929] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:11.430097 1208 log.cc:1199] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/cefcb15849d1431fb50790343f763ba1
I20251214 20:52:11.430434 1208 ts_tablet_manager.cc:1950] T cefcb15849d1431fb50790343f763ba1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:11.432108 1202 ts_tablet_manager.cc:1434] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20251214 20:52:11.432652 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44f51e11041841e98b9a15d380226e4e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:11.433444 880 tablet_service.cc:1552] Processing DeleteTablet for tablet ba2ed363bbab4f1f865b183297c98daa with delete_type TABLET_DATA_DELETED (replaced by tablet 569a02142c71427d91868b02695ae12b) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.433712 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bfb575249b8403991d21c6fc96eefeb with delete_type TABLET_DATA_DELETED (replaced by tablet 2b92b6d5dde2465684d19aee44a83160) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.433943 1202 tablet_bootstrap.cc:492] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.401594 1206 tablet_bootstrap.cc:492] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:11.434753 848 leader_election.cc:343] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 6381cbf05a8c4f84b71f229497fd28ef
I20251214 20:52:11.435891 1202 tablet_bootstrap.cc:654] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.436331 1203 tablet_bootstrap.cc:492] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:11.437042 1320 consensus_queue.cc:1048] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20251214 20:52:11.437826 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6381cbf05a8c4f84b71f229497fd28ef" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:11.438130 1055 raft_consensus.cc:2393] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2e8fd1cb731446d88f8e510c7a52999 in current term 1: Already voted for candidate 074399864fb2470fa9e2ee87c1fe2212 in this term.
I20251214 20:52:11.440081 915 raft_consensus.cc:3060] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.407361 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44f51e11041841e98b9a15d380226e4e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:11.446906 1202 tablet_bootstrap.cc:492] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.447203 1202 ts_tablet_manager.cc:1403] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20251214 20:52:11.449095 1202 raft_consensus.cc:359] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.449196 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_7f018aa7a0aa4ab3895001f6c97c48bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:11.449430 1202 raft_consensus.cc:385] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.449522 1202 raft_consensus.cc:740] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.449774 1202 consensus_queue.cc:260] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:11.450001 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f018aa7a0aa4ab3895001f6c97c48bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.450310 1202 ts_tablet_manager.cc:1434] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251214 20:52:11.450673 1202 tablet_bootstrap.cc:492] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.451902 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7aff79ce154385a73c37b10b9c2430 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.452486 1202 tablet_bootstrap.cc:654] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.454779 777 raft_consensus.cc:2393] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2e8fd1cb731446d88f8e510c7a52999 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:11.458333 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bfb575249b8403991d21c6fc96eefeb with delete_type TABLET_DATA_DELETED (replaced by tablet 2b92b6d5dde2465684d19aee44a83160) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:11.459579 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_ec65ee1eda49435895dab50f78b4ee15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:11.460090 1205 raft_consensus.cc:2272] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:11.460268 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec65ee1eda49435895dab50f78b4ee15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.461759 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_76a2939ccdea4fdeaa1da24713bc3ca7" } modify_external_catalogs: true
I20251214 20:52:11.461947 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_76a2939ccdea4fdeaa1da24713bc3ca7" } modify_external_catalogs: true
I20251214 20:52:11.462868 851 leader_election.cc:304] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:11.463263 1213 raft_consensus.cc:3060] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:11.464443 512 consensus_peers.cc:597] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad -> Peer 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099): Couldn't send request to peer 7c48b8eaf30c4e2caa94a51d087406d4. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:35099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251214 20:52:11.459357 1211 consensus_queue.cc:1048] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20251214 20:52:11.466329 1213 raft_consensus.cc:2749] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:11.467880 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c344296be0549e88d4811103eb6fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.468578 851 leader_election.cc:304] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:11.469632 1213 raft_consensus.cc:2749] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:11.485342 1202 tablet_bootstrap.cc:492] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.485649 1202 ts_tablet_manager.cc:1403] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.022s
I20251214 20:52:11.486554 1202 raft_consensus.cc:359] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.486853 1202 raft_consensus.cc:385] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.486985 1202 raft_consensus.cc:740] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.487350 1202 consensus_queue.cc:260] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.490406 1202 ts_tablet_manager.cc:1434] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:11.492074 1202 tablet_bootstrap.cc:492] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.494092 1202 tablet_bootstrap.cc:654] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.498723 1205 ts_tablet_manager.cc:1916] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.501804 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bfb575249b8403991d21c6fc96eefeb with delete_type TABLET_DATA_DELETED (replaced by tablet 2b92b6d5dde2465684d19aee44a83160) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.503681 1211 consensus_queue.cc:1048] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
I20251214 20:52:11.505208 1202 tablet_bootstrap.cc:492] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.505404 1202 ts_tablet_manager.cc:1403] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20251214 20:52:11.506865 1202 raft_consensus.cc:359] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.507087 1202 raft_consensus.cc:385] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.507187 1202 raft_consensus.cc:740] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.507434 1202 consensus_queue.cc:260] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.507890 1202 ts_tablet_manager.cc:1434] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:11.508194 1202 tablet_bootstrap.cc:492] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.512423 1202 tablet_bootstrap.cc:654] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.522135 1206 tablet_bootstrap.cc:654] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.525941 1207 raft_consensus.cc:2243] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:11.529911 1207 pending_rounds.cc:70] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:11.530215 1207 pending_rounds.cc:77] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493695198945280 op_type: NO_OP noop_request { }
I20251214 20:52:11.530380 1207 raft_consensus.cc:2889] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:11.530511 1207 raft_consensus.cc:2272] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.531507 1207 ts_tablet_manager.cc:1916] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.540169 1207 ts_tablet_manager.cc:1929] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:11.540462 1207 log.cc:1199] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/0a0ca65d369a45958c6dac539964114b
I20251214 20:52:11.541028 1207 ts_tablet_manager.cc:1950] T 0a0ca65d369a45958c6dac539964114b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:11.541867 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 0a0ca65d369a45958c6dac539964114b (table default.Table_28368c3015e84594800ff8a110b0b4c3 [id=45ca58c541924d98bf3a7f04078ac204]) successfully deleted
W20251214 20:52:11.542722 510 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 0a0ca65d369a45958c6dac539964114b because the tablet was not found. No further retry: Not found: Tablet not found: 0a0ca65d369a45958c6dac539964114b
W20251214 20:52:11.543486 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet dc442726ec0c456290ecf2892d0df957 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc442726ec0c456290ecf2892d0df957 already in progress: creating tablet
W20251214 20:52:11.544189 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 5ba2d34a6d2645bd97fe85fed952692b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ba2d34a6d2645bd97fe85fed952692b already in progress: creating tablet
I20251214 20:52:11.544673 1207 tablet_replica.cc:333] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:11.546366 1207 raft_consensus.cc:2243] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:11.553375 1203 tablet_bootstrap.cc:654] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.560738 882 tablet_service.cc:1505] Processing CreateTablet for tablet c16895ca08d94b56b5c4bc9352e330ef (DEFAULT_TABLE table=default.Table_9f6bb794724346069627cd2f14a5cd1b [id=4e30619d28324239bc069a5ce053b167]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.561331 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c16895ca08d94b56b5c4bc9352e330ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.564086 880 tablet_service.cc:1505] Processing CreateTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 (DEFAULT_TABLE table=default.Table_01dcfe83d7854b4e9b8d6f62a557a0fe [id=d180e3a95c0840589c0aeec13ef1a6ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.564625 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60ae3db968f6412aa0c3952fafe65f88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.567553 888 tablet_service.cc:1505] Processing CreateTablet for tablet 7d4626d02ca548f3bf5a1a9142600bff (DEFAULT_TABLE table=default.Table_997191dc162f47a4b9e3b05289f6cb48 [id=d402227da94f418bbed00186b8ac48c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.567961 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4626d02ca548f3bf5a1a9142600bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.571497 889 tablet_service.cc:1505] Processing CreateTablet for tablet 9523d72fd91445539d67e1eac2a80afd (DEFAULT_TABLE table=default.Table_76a2939ccdea4fdeaa1da24713bc3ca7 [id=edb7e58d7c0b4ddea05ba0139f321042]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.574084 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 (DEFAULT_TABLE table=default.Table_01dcfe83d7854b4e9b8d6f62a557a0fe [id=d180e3a95c0840589c0aeec13ef1a6ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.574625 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60ae3db968f6412aa0c3952fafe65f88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.579701 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 9523d72fd91445539d67e1eac2a80afd (DEFAULT_TABLE table=default.Table_76a2939ccdea4fdeaa1da24713bc3ca7 [id=edb7e58d7c0b4ddea05ba0139f321042]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.580093 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9523d72fd91445539d67e1eac2a80afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.581996 886 tablet_service.cc:1505] Processing CreateTablet for tablet 4a625fcb95324466a12ab118bdb859f1 (DEFAULT_TABLE table=default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.584908 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 4a625fcb95324466a12ab118bdb859f1 (DEFAULT_TABLE table=default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.585428 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a625fcb95324466a12ab118bdb859f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.587457 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9523d72fd91445539d67e1eac2a80afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.588119 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a625fcb95324466a12ab118bdb859f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.588294 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet cefcb15849d1431fb50790343f763ba1 (table default.Table_af25ec2aae8a46e48066cfe3f391b504 [id=5661f9fb5cc24abfa05d40d479b6fc1f]) successfully deleted
I20251214 20:52:11.588768 1208 tablet_replica.cc:333] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:11.588999 1208 raft_consensus.cc:2243] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:11.589227 1208 raft_consensus.cc:2272] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.605158 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 7d4626d02ca548f3bf5a1a9142600bff (DEFAULT_TABLE table=default.Table_997191dc162f47a4b9e3b05289f6cb48 [id=d402227da94f418bbed00186b8ac48c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.605773 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4626d02ca548f3bf5a1a9142600bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.617013 1031 tablet_service.cc:1505] Processing CreateTablet for tablet c16895ca08d94b56b5c4bc9352e330ef (DEFAULT_TABLE table=default.Table_9f6bb794724346069627cd2f14a5cd1b [id=4e30619d28324239bc069a5ce053b167]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.619197 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c16895ca08d94b56b5c4bc9352e330ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.623260 1208 ts_tablet_manager.cc:1916] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.575208 748 tablet_service.cc:1505] Processing CreateTablet for tablet 7d4626d02ca548f3bf5a1a9142600bff (DEFAULT_TABLE table=default.Table_997191dc162f47a4b9e3b05289f6cb48 [id=d402227da94f418bbed00186b8ac48c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.626662 1207 raft_consensus.cc:2272] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.626943 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4626d02ca548f3bf5a1a9142600bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.575358 749 tablet_service.cc:1505] Processing CreateTablet for tablet c16895ca08d94b56b5c4bc9352e330ef (DEFAULT_TABLE table=default.Table_9f6bb794724346069627cd2f14a5cd1b [id=4e30619d28324239bc069a5ce053b167]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.631477 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c16895ca08d94b56b5c4bc9352e330ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.633363 1207 ts_tablet_manager.cc:1916] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.575618 745 tablet_service.cc:1505] Processing CreateTablet for tablet 9523d72fd91445539d67e1eac2a80afd (DEFAULT_TABLE table=default.Table_76a2939ccdea4fdeaa1da24713bc3ca7 [id=edb7e58d7c0b4ddea05ba0139f321042]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.635572 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9523d72fd91445539d67e1eac2a80afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.636487 1206 tablet_bootstrap.cc:492] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:11.636711 1217 raft_consensus.cc:493] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.636897 1217 raft_consensus.cc:515] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.637696 1217 leader_election.cc:290] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.575731 742 tablet_service.cc:1505] Processing CreateTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 (DEFAULT_TABLE table=default.Table_01dcfe83d7854b4e9b8d6f62a557a0fe [id=d180e3a95c0840589c0aeec13ef1a6ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.639845 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60ae3db968f6412aa0c3952fafe65f88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.640051 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35101a6997e24040994661756d9649d4" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:11.640288 915 raft_consensus.cc:2468] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
W20251214 20:52:11.641945 523 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84a0237e4ee34f94821d8647a400575b consensus state has no leader
I20251214 20:52:11.575973 756 tablet_service.cc:1505] Processing CreateTablet for tablet 4a625fcb95324466a12ab118bdb859f1 (DEFAULT_TABLE table=default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:11.643927 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a625fcb95324466a12ab118bdb859f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:11.644043 712 leader_election.cc:304] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:11.644420 1217 raft_consensus.cc:2804] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:11.644555 1217 raft_consensus.cc:493] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:11.644644 1217 raft_consensus.cc:3060] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.647292 1217 raft_consensus.cc:515] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.648144 1217 leader_election.cc:290] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.653592 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35101a6997e24040994661756d9649d4" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:11.653930 915 raft_consensus.cc:3060] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.657194 915 raft_consensus.cc:2468] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:11.659160 712 leader_election.cc:304] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:11.659595 1217 raft_consensus.cc:2804] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:11.660369 1217 raft_consensus.cc:697] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:11.660777 1217 consensus_queue.cc:237] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.665437 888 tablet_service.cc:1552] Processing DeleteTablet for tablet a05f1ec28fb24508ae937c70aaaefffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.666862 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a0ca65d369a45958c6dac539964114b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:11.666893 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 0a0ca65d369a45958c6dac539964114b from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:11.668027 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_d677f51980c54afd9712bf047e17f49e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:11.669054 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d677f51980c54afd9712bf047e17f49e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.670025 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35101a6997e24040994661756d9649d4" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:11.670687 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35101a6997e24040994661756d9649d4" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:11.670763 882 tablet_service.cc:1552] Processing DeleteTablet for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
W20251214 20:52:11.671612 713 leader_election.cc:343] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: Tablet not RUNNING: STOPPED
W20251214 20:52:11.672005 713 leader_election.cc:343] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: Tablet not RUNNING: STOPPED
I20251214 20:52:11.672606 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet ba2ed363bbab4f1f865b183297c98daa with delete_type TABLET_DATA_DELETED (replaced by tablet 569a02142c71427d91868b02695ae12b) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.675868 1206 ts_tablet_manager.cc:1403] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.235s user 0.002s sys 0.001s
I20251214 20:52:11.677330 1206 raft_consensus.cc:359] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.678965 1202 tablet_bootstrap.cc:492] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.681031 1202 ts_tablet_manager.cc:1403] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.173s user 0.021s sys 0.048s
I20251214 20:52:11.681994 1202 raft_consensus.cc:359] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.682271 1202 raft_consensus.cc:385] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.682394 1202 raft_consensus.cc:740] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.682737 1202 consensus_queue.cc:260] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.683337 1202 ts_tablet_manager.cc:1434] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:11.683658 1202 tablet_bootstrap.cc:492] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:11.686676 521 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 472e4a5990e64d02b1d52dc6e86d7190 consensus state has no leader
I20251214 20:52:11.687851 1202 tablet_bootstrap.cc:654] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.694550 521 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 358d5c6b594147c8ba136b4b0732fae1 (table default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]). Expected version 1 got 0
I20251214 20:52:11.695495 1202 tablet_bootstrap.cc:492] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.696305 1202 ts_tablet_manager.cc:1403] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20251214 20:52:11.697386 1202 raft_consensus.cc:359] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.698665 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_cb89b84fb9ec4a0d89a3fd65d538725e" } new_table_name: "default.Table_520f27438b664effa96397e5bf61379d" modify_external_catalogs: true
I20251214 20:52:11.699021 1363 tablet.cc:1722] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:11.699560 1202 raft_consensus.cc:385] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.700704 1202 raft_consensus.cc:740] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.701303 1202 consensus_queue.cc:260] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.705811 1202 ts_tablet_manager.cc:1434] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:11.706475 1202 tablet_bootstrap.cc:492] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.708966 1202 tablet_bootstrap.cc:654] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.714329 1208 ts_tablet_manager.cc:1929] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:11.717368 1320 consensus_queue.cc:1048] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:11.702700 742 tablet_service.cc:1552] Processing DeleteTablet for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:11.718964 1206 raft_consensus.cc:385] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.719262 1206 raft_consensus.cc:740] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.719712 1206 consensus_queue.cc:260] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.734300 1349 consensus_queue.cc:1048] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20251214 20:52:11.721920 1364 tablet.cc:1722] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251214 20:52:11.737699 991 consensus_peers.cc:597] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6cc54be0bc6d4f259967fbe0a5023f82. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:11.743086 1205 ts_tablet_manager.cc:1929] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:11.743499 1205 log.cc:1199] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/35101a6997e24040994661756d9649d4
I20251214 20:52:11.743986 1203 tablet_bootstrap.cc:492] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:11.744361 1205 ts_tablet_manager.cc:1950] T 35101a6997e24040994661756d9649d4 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:11.745301 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 83) took 1129 ms. Trace:
I20251214 20:52:11.752326 1206 ts_tablet_manager.cc:1434] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.076s user 0.000s sys 0.002s
I20251214 20:52:11.745519 1205 rpcz_store.cc:276] 1214 20:52:10.615826 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.615888 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:11.320973 (+705085us) ts_tablet_manager.cc:1151] Deleting tablet 35101a6997e24040994661756d9649d4
1214 20:52:11.320990 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:11.703965 (+382975us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:11.739926 (+ 35961us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:11.745250 (+ 5324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":692208}
I20251214 20:52:11.755671 1205 tablet_replica.cc:333] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:11.755903 1205 raft_consensus.cc:2243] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:11.756363 1205 raft_consensus.cc:2272] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.756340 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 35101a6997e24040994661756d9649d4 (table default.Table_452ec7356ff74de288f4a0fdfbc09dca [id=2db316c4d7814e6091fdcf30b0b4aa0e]) successfully deleted
I20251214 20:52:11.757009 1205 ts_tablet_manager.cc:1916] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.757488 1203 ts_tablet_manager.cc:1403] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.324s user 0.004s sys 0.013s
I20251214 20:52:11.758453 1203 raft_consensus.cc:359] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.758659 1203 raft_consensus.cc:385] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.758733 1203 raft_consensus.cc:740] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.759033 1206 tablet_bootstrap.cc:492] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:11.758986 1203 consensus_queue.cc:260] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.764395 1203 ts_tablet_manager.cc:1434] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:11.764753 1203 tablet_bootstrap.cc:492] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:11.721997 1202 tablet_bootstrap.cc:492] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.775516 1202 ts_tablet_manager.cc:1403] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.000s
I20251214 20:52:11.776193 1213 raft_consensus.cc:493] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.776353 1213 raft_consensus.cc:515] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.776420 1202 raft_consensus.cc:359] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:11.776551 1202 raft_consensus.cc:385] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.776638 1202 raft_consensus.cc:740] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.776834 1202 consensus_queue.cc:260] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:11.777067 1213 leader_election.cc:290] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.777390 1202 ts_tablet_manager.cc:1434] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:11.777676 1202 tablet_bootstrap.cc:492] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:11.778133 848 leader_election.cc:343] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 2a0eb9ab79d9428997c6f66790d6b1a3
I20251214 20:52:11.780673 1202 tablet_bootstrap.cc:654] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.781567 1208 log.cc:1199] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ac19d32a31e7405b9f16566215d6b2cd
I20251214 20:52:11.782955 522 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 358d5c6b594147c8ba136b4b0732fae1 (table default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]). Expected version 1 got 0
I20251214 20:52:11.784035 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_e744815cb34441df8b96e5ab9527681e" } new_table_name: "default.Table_5000227c87004ed7a604c72840d805da" modify_external_catalogs: true
I20251214 20:52:11.784596 1206 tablet_bootstrap.cc:654] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:11.788184 851 leader_election.cc:343] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 2a0eb9ab79d9428997c6f66790d6b1a3
I20251214 20:52:11.788365 1203 tablet_bootstrap.cc:654] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.788641 851 leader_election.cc:304] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:11.789292 1342 raft_consensus.cc:2749] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:11.789796 1208 ts_tablet_manager.cc:1950] T ac19d32a31e7405b9f16566215d6b2cd P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:11.793955 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c344296be0549e88d4811103eb6fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.794965 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_ba612520b0be474db6573ce754044c8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:11.796072 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba612520b0be474db6573ce754044c8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.796766 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 2842f1dc0c374d24946f90dac1cf8aa5
I20251214 20:52:11.797523 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 0c344296be0549e88d4811103eb6fee5 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:11.800323 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:11.801258 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.806198 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 6bd6856b800247de896f16eee66f2827 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:11.792991 1342 raft_consensus.cc:493] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.798466 1202 tablet_bootstrap.cc:492] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.810894 1202 ts_tablet_manager.cc:1403] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.004s
I20251214 20:52:11.810801 1342 raft_consensus.cc:515] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.811894 1202 raft_consensus.cc:359] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.812155 1202 raft_consensus.cc:385] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.812247 1202 raft_consensus.cc:740] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.812494 1202 consensus_queue.cc:260] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.812861 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a0ca65d369a45958c6dac539964114b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:11.812968 1202 ts_tablet_manager.cc:1434] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:11.813140 1055 raft_consensus.cc:2393] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2e8fd1cb731446d88f8e510c7a52999 in current term 1: Already voted for candidate 074399864fb2470fa9e2ee87c1fe2212 in this term.
I20251214 20:52:11.813251 1202 tablet_bootstrap.cc:492] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.814745 1202 tablet_bootstrap.cc:654] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.811906 1342 leader_election.cc:290] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.800377 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:43856
W20251214 20:52:11.817282 848 leader_election.cc:343] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 0a0ca65d369a45958c6dac539964114b
I20251214 20:52:11.817446 848 leader_election.cc:304] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:11.817770 1342 raft_consensus.cc:3060] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.821166 1342 raft_consensus.cc:2749] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:11.824963 1205 ts_tablet_manager.cc:1929] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:11.825143 1205 log.cc:1199] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/6381cbf05a8c4f84b71f229497fd28ef
I20251214 20:52:11.825521 1205 ts_tablet_manager.cc:1950] T 6381cbf05a8c4f84b71f229497fd28ef P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:11.828586 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ac19d32a31e7405b9f16566215d6b2cd (table default.Table_8090a0016b2248f6b8fa395e4f676734 [id=ac9c660a341b4fe6abedd5343196a30a]) successfully deleted
I20251214 20:52:11.828617 1202 tablet_bootstrap.cc:492] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:11.828892 1202 ts_tablet_manager.cc:1403] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20251214 20:52:11.829286 1208 tablet_replica.cc:333] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:11.829622 1208 raft_consensus.cc:2243] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:11.829917 1208 raft_consensus.cc:2272] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.829954 1202 raft_consensus.cc:359] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.830181 1202 raft_consensus.cc:385] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.830286 1202 raft_consensus.cc:740] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.830789 1202 consensus_queue.cc:260] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.832180 1202 ts_tablet_manager.cc:1434] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:11.832516 1202 tablet_bootstrap.cc:492] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:11.835075 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 87) took 1183 ms. Trace:
I20251214 20:52:11.835258 1205 rpcz_store.cc:276] 1214 20:52:10.645877 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.645939 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:11.755436 (+1109497us) ts_tablet_manager.cc:1151] Deleting tablet 6381cbf05a8c4f84b71f229497fd28ef
1214 20:52:11.755446 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:11.762465 (+ 7019us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:11.824782 (+ 62317us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:11.829491 (+ 4709us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109299}
I20251214 20:52:11.835819 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 6381cbf05a8c4f84b71f229497fd28ef (table default.Table_035a23623ce242298ab89c51024d7a09 [id=cd7616f6ff3244ec8628da5102e51570]) successfully deleted
I20251214 20:52:11.836397 1202 tablet_bootstrap.cc:654] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.836644 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 93) took 1096 ms. Trace:
I20251214 20:52:11.836748 1205 rpcz_store.cc:276] 1214 20:52:10.739634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.739711 (+ 77us) service_pool.cc:225] Handling call
1214 20:52:11.836296 (+1096585us) ts_tablet_manager.cc:1151] Deleting tablet dc442726ec0c456290ecf2892d0df957
1214 20:52:11.836310 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:11.836470 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1092708}
I20251214 20:52:11.836858 1342 raft_consensus.cc:493] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.837144 1342 raft_consensus.cc:515] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:11.837340 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet dc442726ec0c456290ecf2892d0df957 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc442726ec0c456290ecf2892d0df957 already in progress: creating tablet
I20251214 20:52:11.838471 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c344296be0549e88d4811103eb6fee5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:11.838726 1055 raft_consensus.cc:2468] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:11.839214 851 leader_election.cc:304] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:11.839563 1205 tablet_replica.cc:333] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:11.839563 1213 raft_consensus.cc:2804] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:11.839674 1213 raft_consensus.cc:493] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:11.839757 1213 raft_consensus.cc:3060] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.839768 1205 raft_consensus.cc:2243] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:11.839962 1205 raft_consensus.cc:2272] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.840596 1205 ts_tablet_manager.cc:1916] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.844223 1342 leader_election.cc:290] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.845175 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c344296be0549e88d4811103eb6fee5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:11.845976 1213 raft_consensus.cc:515] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.847708 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c344296be0549e88d4811103eb6fee5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:11.848019 1055 raft_consensus.cc:3060] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.851436 1213 leader_election.cc:290] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:11.852144 848 leader_election.cc:343] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:11.852466 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c344296be0549e88d4811103eb6fee5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:11.852913 1055 raft_consensus.cc:2468] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:11.853518 851 leader_election.cc:304] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:11.854017 848 leader_election.cc:343] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:11.854465 1213 raft_consensus.cc:2804] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:11.854660 1213 raft_consensus.cc:697] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:11.855051 1213 consensus_queue.cc:237] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.830799 1208 ts_tablet_manager.cc:1916] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.858547 1206 tablet_bootstrap.cc:492] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:11.858789 1206 ts_tablet_manager.cc:1403] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.000s
I20251214 20:52:11.859180 1207 ts_tablet_manager.cc:1929] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:11.859553 1207 log.cc:1199] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ba2ed363bbab4f1f865b183297c98daa
I20251214 20:52:11.862089 1205 ts_tablet_manager.cc:1929] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:11.862238 1205 log.cc:1199] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/0a0ca65d369a45958c6dac539964114b
I20251214 20:52:11.860489 1206 raft_consensus.cc:359] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.862560 1206 raft_consensus.cc:385] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.862643 1205 ts_tablet_manager.cc:1950] T 0a0ca65d369a45958c6dac539964114b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:11.862680 1206 raft_consensus.cc:740] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.863008 1206 consensus_queue.cc:260] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.863744 1207 ts_tablet_manager.cc:1950] T ba2ed363bbab4f1f865b183297c98daa P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
W20251214 20:52:11.867278 510 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42865, user_credentials={real_user=slave}} blocked reactor thread for 52408.7us
I20251214 20:52:11.869880 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 9523d72fd91445539d67e1eac2a80afd
I20251214 20:52:11.872345 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 9523d72fd91445539d67e1eac2a80afd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.876633 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 9523d72fd91445539d67e1eac2a80afd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.883903 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 91) took 1144 ms. Trace:
I20251214 20:52:11.884114 1205 rpcz_store.cc:276] 1214 20:52:10.739394 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.739837 (+ 443us) service_pool.cc:225] Handling call
1214 20:52:11.839455 (+1099618us) ts_tablet_manager.cc:1151] Deleting tablet 0a0ca65d369a45958c6dac539964114b
1214 20:52:11.839477 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:11.858095 (+ 18618us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:11.861906 (+ 3811us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:11.883829 (+ 21923us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1095561}
I20251214 20:52:11.884758 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 0a0ca65d369a45958c6dac539964114b (table default.Table_28368c3015e84594800ff8a110b0b4c3 [id=45ca58c541924d98bf3a7f04078ac204]) successfully deleted
I20251214 20:52:11.886190 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 94) took 1146 ms. Trace:
I20251214 20:52:11.886868 1205 rpcz_store.cc:276] 1214 20:52:10.739748 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.740290 (+ 542us) service_pool.cc:225] Handling call
1214 20:52:11.885950 (+1145660us) ts_tablet_manager.cc:1151] Deleting tablet 5ba2d34a6d2645bd97fe85fed952692b
1214 20:52:11.885969 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:11.886140 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138480}
W20251214 20:52:11.887980 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 5ba2d34a6d2645bd97fe85fed952692b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ba2d34a6d2645bd97fe85fed952692b already in progress: creating tablet
W20251214 20:52:11.888789 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 649f5bd67cbb4f61b7066269f0bf9266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 649f5bd67cbb4f61b7066269f0bf9266 already in progress: creating tablet
I20251214 20:52:11.888965 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet ed1d007f1ca24b7291262708ddfca3c3
I20251214 20:52:11.888795 518 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet c16895ca08d94b56b5c4bc9352e330ef
I20251214 20:52:11.889638 520 master_service.cc:954] ReplaceTablet: received request to replace tablet b1df0dad2fb040a5a12e0401e00c87db from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:11.890018 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet f3a2143b5508445492b970d2588cf1cf
I20251214 20:52:11.891652 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_ade67d7e10e04c6fb0cfc134947a559d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:11.891819 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_4b9e6d9e76414e6da9e6767b3463fa10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:11.892338 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b9e6d9e76414e6da9e6767b3463fa10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.892840 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4efe28833e3d4ad8b4e1bb8966338a62" } new_table_name: "default.Table_a519bf6d1ab8419784be3772e899869d" modify_external_catalogs: true
I20251214 20:52:11.894424 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet ed1d007f1ca24b7291262708ddfca3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:11.892356 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ade67d7e10e04c6fb0cfc134947a559d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.894519 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a2143b5508445492b970d2588cf1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.895103 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet c16895ca08d94b56b5c4bc9352e330ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:11.900976 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_6c56ebc90d1a4b92bc80b11ed334a791"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:11.901602 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c56ebc90d1a4b92bc80b11ed334a791 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.905563 888 tablet_service.cc:1552] Processing DeleteTablet for tablet ed1d007f1ca24b7291262708ddfca3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.905881 880 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a2143b5508445492b970d2588cf1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.906096 882 tablet_service.cc:1552] Processing DeleteTablet for tablet c16895ca08d94b56b5c4bc9352e330ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:11.907162 1396 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:35099 every 8 connection(s)
I20251214 20:52:11.907568 1181 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:35099
I20251214 20:52:11.910063 1181 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-2/data/info.pb
I20251214 20:52:11.870131 1206 ts_tablet_manager.cc:1434] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20251214 20:52:11.911418 1206 tablet_bootstrap.cc:492] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:11.913169 1203 tablet_bootstrap.cc:492] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:11.913367 1203 ts_tablet_manager.cc:1403] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.149s user 0.010s sys 0.014s
I20251214 20:52:11.913587 1206 tablet_bootstrap.cc:654] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.876789 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 9523d72fd91445539d67e1eac2a80afd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:11.914769 1203 raft_consensus.cc:359] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.914961 1203 raft_consensus.cc:385] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.915051 1203 raft_consensus.cc:740] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.899597 1217 raft_consensus.cc:493] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.904577 756 tablet_service.cc:1552] Processing DeleteTablet for tablet c16895ca08d94b56b5c4bc9352e330ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:11.904819 748 tablet_service.cc:1552] Processing DeleteTablet for tablet ed1d007f1ca24b7291262708ddfca3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:11.904976 749 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a2143b5508445492b970d2588cf1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:11.905090 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c344296be0549e88d4811103eb6fee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:11.915386 1203 consensus_queue.cc:260] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.916378 1203 ts_tablet_manager.cc:1434] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251214 20:52:11.918251 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu as pid 1181
I20251214 20:52:11.921227 1217 raft_consensus.cc:515] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.922142 1217 leader_election.cc:290] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.922816 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c344296be0549e88d4811103eb6fee5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:11.923103 1055 raft_consensus.cc:2393] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate a2e8fd1cb731446d88f8e510c7a52999 in this term.
I20251214 20:52:11.923080 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c344296be0549e88d4811103eb6fee5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:11.924050 713 leader_election.cc:304] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:11.924135 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 92) took 1130 ms. Trace:
I20251214 20:52:11.924271 1207 rpcz_store.cc:276] 1214 20:52:10.783406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.783477 (+ 71us) service_pool.cc:225] Handling call
1214 20:52:11.544593 (+761116us) ts_tablet_manager.cc:1151] Deleting tablet ba2ed363bbab4f1f865b183297c98daa
1214 20:52:11.544603 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:11.771704 (+227101us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:11.834257 (+ 62553us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:11.914215 (+ 79958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":760846}
I20251214 20:52:11.925704 1217 raft_consensus.cc:3060] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.928735 1217 raft_consensus.cc:2749] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:11.928776 1207 tablet_replica.cc:333] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:11.928962 1207 raft_consensus.cc:2243] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:11.929212 1207 pending_rounds.cc:70] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:11.929281 1207 pending_rounds.cc:77] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493697685057536 op_type: NO_OP noop_request { }
I20251214 20:52:11.929436 1207 raft_consensus.cc:2889] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:11.929529 1207 raft_consensus.cc:2272] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.930317 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ba2ed363bbab4f1f865b183297c98daa (table default.Table_ad4bec26fc854d67b2ec37f504851a01 [id=17c89c03be10405f834504b803e594c7]) successfully deleted
W20251214 20:52:11.936297 521 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd88d1e95ee04ef596863e1ae3687de0 consensus state has no leader
I20251214 20:52:11.938378 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_1a95326a5ce4436ca97f62d32966ec22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:11.939733 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_2f947b9f382942e082ea4909657a8f34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:11.940254 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f947b9f382942e082ea4909657a8f34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.944017 1206 tablet_bootstrap.cc:492] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:11.944234 1206 ts_tablet_manager.cc:1403] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.003s
I20251214 20:52:11.945537 1206 raft_consensus.cc:359] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.945761 1206 raft_consensus.cc:385] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.945827 1206 raft_consensus.cc:740] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.946060 1206 consensus_queue.cc:260] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.948153 1206 ts_tablet_manager.cc:1434] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20251214 20:52:11.948328 522 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 25974e5db555447baf5799775d15cd14 consensus state has no leader
I20251214 20:52:11.948489 1206 tablet_bootstrap.cc:492] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:11.948676 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 358d5c6b594147c8ba136b4b0732fae1 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:11.951099 1207 ts_tablet_manager.cc:1916] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:11.952270 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a95326a5ce4436ca97f62d32966ec22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:11.953805 1206 tablet_bootstrap.cc:654] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.895910 1205 tablet_replica.cc:333] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:11.962486 1206 tablet_bootstrap.cc:492] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:11.962719 1206 ts_tablet_manager.cc:1403] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20251214 20:52:11.962949 1205 raft_consensus.cc:2243] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:11.963639 1205 raft_consensus.cc:2272] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:11.963728 1206 raft_consensus.cc:359] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.963953 1206 raft_consensus.cc:385] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.964082 1206 raft_consensus.cc:740] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.964401 1206 consensus_queue.cc:260] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.964010 1203 tablet_bootstrap.cc:492] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:11.967111 1203 tablet_bootstrap.cc:654] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.970429 1206 ts_tablet_manager.cc:1434] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20251214 20:52:11.971021 1203 tablet_bootstrap.cc:492] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:11.971225 1203 ts_tablet_manager.cc:1403] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20251214 20:52:11.972121 1203 raft_consensus.cc:359] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.972362 1203 raft_consensus.cc:385] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.972450 1203 raft_consensus.cc:740] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.972734 1203 consensus_queue.cc:260] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.973318 1206 tablet_bootstrap.cc:492] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:11.973681 1203 ts_tablet_manager.cc:1434] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:11.973639 1213 raft_consensus.cc:493] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:11.973836 1213 raft_consensus.cc:515] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.973990 1203 tablet_bootstrap.cc:492] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:11.974162 1205 ts_tablet_manager.cc:1916] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:11.974913 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f7aff79ce154385a73c37b10b9c2430" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:11.975140 777 raft_consensus.cc:2468] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:11.975323 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f7aff79ce154385a73c37b10b9c2430" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:11.975523 1055 raft_consensus.cc:2468] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:11.975646 848 leader_election.cc:304] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:11.975953 1203 tablet_bootstrap.cc:654] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.976114 1342 raft_consensus.cc:2804] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:11.976250 1342 raft_consensus.cc:493] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:11.976336 1342 raft_consensus.cc:3060] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.977406 1206 tablet_bootstrap.cc:654] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:11.979224 1213 leader_election.cc:290] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:11.980036 1342 raft_consensus.cc:515] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:11.981254 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f7aff79ce154385a73c37b10b9c2430" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:11.981478 777 raft_consensus.cc:3060] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.983903 1206 tablet_bootstrap.cc:492] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:11.984122 1206 ts_tablet_manager.cc:1403] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20251214 20:52:11.984498 777 raft_consensus.cc:2468] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:11.985074 848 leader_election.cc:304] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:11.986384 1206 raft_consensus.cc:359] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:11.986532 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f7aff79ce154385a73c37b10b9c2430" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:11.986646 1206 raft_consensus.cc:385] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:11.986727 1206 raft_consensus.cc:740] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:11.986768 1055 raft_consensus.cc:3060] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:11.987994 1206 consensus_queue.cc:260] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:11.989647 1205 ts_tablet_manager.cc:1929] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:11.989765 1055 raft_consensus.cc:2468] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:11.989835 1205 log.cc:1199] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/6cc54be0bc6d4f259967fbe0a5023f82
I20251214 20:52:11.990273 1205 ts_tablet_manager.cc:1950] T 6cc54be0bc6d4f259967fbe0a5023f82 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:11.990988 1206 ts_tablet_manager.cc:1434] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251214 20:52:11.991330 1206 tablet_bootstrap.cc:492] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:11.991715 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]) successfully deleted
I20251214 20:52:11.992349 1213 raft_consensus.cc:2804] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:11.992748 1342 leader_election.cc:290] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:11.999276 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e2ee5cc95304fc1a80ed4d31838ac38 already in progress: creating tablet
W20251214 20:52:12.000195 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 6b4f582d8165439d86d9228c146dc41a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b4f582d8165439d86d9228c146dc41a already in progress: creating tablet
I20251214 20:52:12.000634 1207 ts_tablet_manager.cc:1929] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.000773 1206 tablet_bootstrap.cc:654] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.000829 1207 log.cc:1199] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/35101a6997e24040994661756d9649d4
I20251214 20:52:12.001303 1207 ts_tablet_manager.cc:1950] T 35101a6997e24040994661756d9649d4 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
W20251214 20:52:12.001760 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 63497e77baeb4d328be2421b5732a28a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63497e77baeb4d328be2421b5732a28a already in progress: creating tablet
I20251214 20:52:12.002321 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 100) took 1169 ms. Trace:
I20251214 20:52:12.002375 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 92) took 1262 ms. Trace:
I20251214 20:52:12.002492 1205 rpcz_store.cc:276] 1214 20:52:10.739527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.820073 (+ 80546us) service_pool.cc:225] Handling call
1214 20:52:12.002227 (+1182154us) ts_tablet_manager.cc:1151] Deleting tablet 0a0ca65d369a45958c6dac539964114b
1214 20:52:12.002238 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.002303 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":932157}
I20251214 20:52:12.002466 1207 rpcz_store.cc:276] 1214 20:52:10.832683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.835188 (+ 2505us) service_pool.cc:225] Handling call
1214 20:52:11.925409 (+1090221us) ts_tablet_manager.cc:1151] Deleting tablet 35101a6997e24040994661756d9649d4
1214 20:52:11.925420 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:11.984534 (+ 59114us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.000464 (+ 15930us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.002250 (+ 1786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1089971}
W20251214 20:52:12.002946 513 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 0a0ca65d369a45958c6dac539964114b because the tablet was not found. No further retry: Not found: Tablet not found: 0a0ca65d369a45958c6dac539964114b
I20251214 20:52:12.003002 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 111) took 1113 ms. Trace:
I20251214 20:52:12.003170 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 35101a6997e24040994661756d9649d4 (table default.Table_452ec7356ff74de288f4a0fdfbc09dca [id=2db316c4d7814e6091fdcf30b0b4aa0e]) successfully deleted
W20251214 20:52:12.004148 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 47a67b1b9dc94e498cad91a72e655a75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47a67b1b9dc94e498cad91a72e655a75 already in progress: creating tablet
I20251214 20:52:12.004750 1205 tablet_replica.cc:333] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:12.004978 1205 raft_consensus.cc:2243] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:12.006314 1205 raft_consensus.cc:2272] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.007534 1205 ts_tablet_manager.cc:1916] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.010645 1206 tablet_bootstrap.cc:492] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:12.011479 1206 ts_tablet_manager.cc:1403] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.002s
I20251214 20:52:12.012399 1206 raft_consensus.cc:359] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.012604 1206 raft_consensus.cc:385] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.012696 1206 raft_consensus.cc:740] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.012931 1206 consensus_queue.cc:260] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.013343 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_017c57bf3637493fb9ccb5976322bfe7" } new_table_name: "default.Table_c5d8c1bdf3344ec5a0d409a1abf98b45" modify_external_catalogs: true
I20251214 20:52:12.016805 1206 ts_tablet_manager.cc:1434] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:12.003098 1207 rpcz_store.cc:276] 1214 20:52:10.889793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.889848 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:12.002805 (+1112957us) ts_tablet_manager.cc:1151] Deleting tablet 649f5bd67cbb4f61b7066269f0bf9266
1214 20:52:12.002815 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.002932 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112700}
W20251214 20:52:12.020205 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 649f5bd67cbb4f61b7066269f0bf9266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 649f5bd67cbb4f61b7066269f0bf9266 already in progress: creating tablet
I20251214 20:52:12.020928 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7aff79ce154385a73c37b10b9c2430 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.021095 1207 tablet_replica.cc:333] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:12.021332 1207 raft_consensus.cc:2243] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.021598 1207 raft_consensus.cc:2272] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.022260 1207 ts_tablet_manager.cc:1916] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.022764 1206 tablet_bootstrap.cc:492] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:12.024703 1206 tablet_bootstrap.cc:654] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.027493 1208 ts_tablet_manager.cc:1929] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.030491 1206 tablet_bootstrap.cc:492] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:12.030733 1206 ts_tablet_manager.cc:1403] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20251214 20:52:12.031788 1206 raft_consensus.cc:359] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.032016 1206 raft_consensus.cc:385] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.032099 1206 raft_consensus.cc:740] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.032435 1206 consensus_queue.cc:260] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.034823 1206 ts_tablet_manager.cc:1434] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251214 20:52:12.035228 1206 tablet_bootstrap.cc:492] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:12.039070 1206 tablet_bootstrap.cc:654] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.046245 1397 sys_catalog.cc:263] Verifying existing consensus state
I20251214 20:52:12.048399 1205 ts_tablet_manager.cc:1929] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.048426 1217 raft_consensus.cc:493] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.048599 1205 log.cc:1199] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/7bfb575249b8403991d21c6fc96eefeb
I20251214 20:52:12.048605 1217 raft_consensus.cc:515] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.049121 1205 ts_tablet_manager.cc:1950] T 7bfb575249b8403991d21c6fc96eefeb P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:12.049446 1217 leader_election.cc:290] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.050184 1205 tablet_replica.cc:333] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:12.050415 1205 raft_consensus.cc:2243] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.050601 1205 raft_consensus.cc:2272] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.051218 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 7bfb575249b8403991d21c6fc96eefeb (table default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3]) successfully deleted
I20251214 20:52:12.051273 1205 ts_tablet_manager.cc:1916] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.051692 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "358d5c6b594147c8ba136b4b0732fae1" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.051940 1055 raft_consensus.cc:2468] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:12.052510 713 leader_election.cc:304] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:12.052865 1217 raft_consensus.cc:2804] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:12.052986 1217 raft_consensus.cc:493] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:12.053076 1217 raft_consensus.cc:3060] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.056037 1217 raft_consensus.cc:515] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.058648 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "358d5c6b594147c8ba136b4b0732fae1" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:12.058864 1055 raft_consensus.cc:3060] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.062021 1055 raft_consensus.cc:2468] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:12.062649 713 leader_election.cc:304] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:12.063027 1341 raft_consensus.cc:2804] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:12.065193 1397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: Bootstrap starting.
I20251214 20:52:12.065536 1217 leader_election.cc:290] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.075765 1341 raft_consensus.cc:697] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:12.076313 1341 consensus_queue.cc:237] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.032773 1213 raft_consensus.cc:697] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:12.079418 1213 consensus_queue.cc:237] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.080060 1205 ts_tablet_manager.cc:1929] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.080296 1205 log.cc:1199] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ba2ed363bbab4f1f865b183297c98daa
I20251214 20:52:12.080698 1205 ts_tablet_manager.cc:1950] T ba2ed363bbab4f1f865b183297c98daa P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:12.042479 1408 raft_consensus.cc:493] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.050875 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "358d5c6b594147c8ba136b4b0732fae1" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:12.081176 913 raft_consensus.cc:2468] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:12.057444 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "358d5c6b594147c8ba136b4b0732fae1" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:12.081892 1205 tablet_replica.cc:333] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:12.082084 1205 raft_consensus.cc:2243] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.082084 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ba2ed363bbab4f1f865b183297c98daa (table default.Table_ad4bec26fc854d67b2ec37f504851a01 [id=17c89c03be10405f834504b803e594c7]) successfully deleted
I20251214 20:52:12.082263 915 raft_consensus.cc:3060] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.082262 1205 raft_consensus.cc:2272] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.085213 915 raft_consensus.cc:2468] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:12.086669 1408 raft_consensus.cc:515] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:12.087970 1408 leader_election.cc:290] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:12.088438 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8947ebdc87341ad8b1a01f5dd246172" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.088992 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8947ebdc87341ad8b1a01f5dd246172" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:12.089226 777 raft_consensus.cc:2468] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:12.089337 1202 tablet_bootstrap.cc:492] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:12.089509 1202 ts_tablet_manager.cc:1403] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.257s user 0.052s sys 0.059s
I20251214 20:52:12.089872 848 leader_election.cc:304] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:12.090483 1202 raft_consensus.cc:359] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:12.090529 851 leader_election.cc:343] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.090699 1202 raft_consensus.cc:385] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.090772 1202 raft_consensus.cc:740] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.090988 822 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:12.091018 1202 consensus_queue.cc:260] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.091362 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.091558 1202 ts_tablet_manager.cc:1434] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251214 20:52:12.091868 1202 tablet_bootstrap.cc:492] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:12.092224 1205 ts_tablet_manager.cc:1916] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.093919 1342 raft_consensus.cc:2804] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:12.094054 1342 raft_consensus.cc:493] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:12.094175 1342 raft_consensus.cc:3060] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.094771 1202 tablet_bootstrap.cc:654] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.095929 1207 ts_tablet_manager.cc:1929] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.096076 1207 log.cc:1199] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/a05f1ec28fb24508ae937c70aaaefffe
I20251214 20:52:12.096114 1208 log.cc:1199] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/35101a6997e24040994661756d9649d4
I20251214 20:52:12.097677 1207 ts_tablet_manager.cc:1950] T a05f1ec28fb24508ae937c70aaaefffe P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:12.097870 1342 raft_consensus.cc:515] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:12.098141 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 112) took 1186 ms. Trace:
I20251214 20:52:12.098311 1207 rpcz_store.cc:276] 1214 20:52:10.911977 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.912042 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:12.020963 (+1108921us) ts_tablet_manager.cc:1151] Deleting tablet a05f1ec28fb24508ae937c70aaaefffe
1214 20:52:12.020977 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.055547 (+ 34570us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.095716 (+ 40169us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.098096 (+ 2380us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1108648}
I20251214 20:52:12.098933 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet a05f1ec28fb24508ae937c70aaaefffe (table default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d]) successfully deleted
I20251214 20:52:12.099522 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8947ebdc87341ad8b1a01f5dd246172" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
W20251214 20:52:12.100227 851 leader_election.cc:343] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.101051 1397 log.cc:826] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: Log is configured to *not* fsync() on all Append() calls
I20251214 20:52:12.101461 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8947ebdc87341ad8b1a01f5dd246172" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:12.101688 777 raft_consensus.cc:3060] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.101964 1205 ts_tablet_manager.cc:1929] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.102082 1205 log.cc:1199] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/0c344296be0549e88d4811103eb6fee5
I20251214 20:52:12.096513 1208 ts_tablet_manager.cc:1950] T 35101a6997e24040994661756d9649d4 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:12.102484 1205 ts_tablet_manager.cc:1950] T 0c344296be0549e88d4811103eb6fee5 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:12.103518 1208 tablet_replica.cc:333] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:12.103768 1208 raft_consensus.cc:2243] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.103731 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 35101a6997e24040994661756d9649d4 (table default.Table_452ec7356ff74de288f4a0fdfbc09dca [id=2db316c4d7814e6091fdcf30b0b4aa0e]) successfully deleted
I20251214 20:52:12.103953 1208 raft_consensus.cc:2272] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.104051 1203 tablet_bootstrap.cc:492] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:12.104208 1203 ts_tablet_manager.cc:1403] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.130s user 0.007s sys 0.010s
I20251214 20:52:12.104548 1208 ts_tablet_manager.cc:1916] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.104902 960 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:12.105100 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.105300 1207 tablet_replica.cc:333] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:12.105216 1203 raft_consensus.cc:359] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.105512 1203 raft_consensus.cc:385] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.105540 1207 raft_consensus.cc:2243] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.105610 1203 raft_consensus.cc:740] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.105875 777 raft_consensus.cc:2468] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:12.105908 1203 consensus_queue.cc:260] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.106559 1207 raft_consensus.cc:2272] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.106642 848 leader_election.cc:304] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:12.107632 1342 leader_election.cc:290] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:12.108057 1342 raft_consensus.cc:2804] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:12.108448 1342 raft_consensus.cc:697] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:12.108477 1207 ts_tablet_manager.cc:1916] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.108749 1203 ts_tablet_manager.cc:1434] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20251214 20:52:12.108873 1342 consensus_queue.cc:237] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:12.109470 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 0c344296be0549e88d4811103eb6fee5 (table default.Table_3c3f2d9e5c2b47ed8038a7402612b093 [id=163e1e9d48bc4f51b308383354454363]) successfully deleted
W20251214 20:52:12.113698 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2842f1dc0c374d24946f90dac1cf8aa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2842f1dc0c374d24946f90dac1cf8aa5 already in progress: creating tablet
I20251214 20:52:12.110708 1203 tablet_bootstrap.cc:492] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:12.115705 1202 tablet_bootstrap.cc:492] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:12.115929 1202 ts_tablet_manager.cc:1403] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20251214 20:52:12.116972 1202 raft_consensus.cc:359] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.117204 1202 raft_consensus.cc:385] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.117303 1202 raft_consensus.cc:740] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.117738 1202 consensus_queue.cc:260] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.118445 1202 ts_tablet_manager.cc:1434] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:12.118832 1202 tablet_bootstrap.cc:492] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:12.119491 525 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b (table default.Table_4efe28833e3d4ad8b4e1bb8966338a62 [id=5cbe2438b4294442a9778bf1f641b60e]). Expected version 1 got 0
I20251214 20:52:12.122218 1206 tablet_bootstrap.cc:492] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:12.122428 1206 ts_tablet_manager.cc:1403] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.009s
W20251214 20:52:12.124338 523 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b consensus state has no leader
I20251214 20:52:12.124368 1206 raft_consensus.cc:359] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.124612 1206 raft_consensus.cc:385] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.124698 1206 raft_consensus.cc:740] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.124727 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 44f51e11041841e98b9a15d380226e4e from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.124949 1206 consensus_queue.cc:260] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.126497 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.127576 882 tablet_service.cc:1505] Processing CreateTablet for tablet 88b9ca42a5ce4572aebe2a24f4758b12 (DEFAULT_TABLE table=default.Table_5242f3b282284ab7aff8525106732d53 [id=0d173bfba2b343dbac6a02a5ff84c828]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.129694 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b9ca42a5ce4572aebe2a24f4758b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.127980 880 tablet_service.cc:1505] Processing CreateTablet for tablet afd51f330eb649b998a22a343d993b30 (DEFAULT_TABLE table=default.Table_cae5b2188af54d068db115a0745ee866 [id=f4157d7b9fc848a8b7fa08742b456167]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.134191 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd51f330eb649b998a22a343d993b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.128626 889 tablet_service.cc:1505] Processing CreateTablet for tablet fe4086fc7a4f4db0b7541f06c9833aab (DEFAULT_TABLE table=default.Table_8076bc8b31ba411faf089f314e6ed2ef [id=25016f04bb864f92a1cdfb6ceaa9fba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.135537 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe4086fc7a4f4db0b7541f06c9833aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.137208 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 52daf1204bf142e7b82852bdec6582fb (DEFAULT_TABLE table=default.Table_bff6195ce6594ee9b4a55eb6058dcb63 [id=a0e13ae34009499d91b4461520c292c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.137645 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52daf1204bf142e7b82852bdec6582fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.137728 1032 tablet_service.cc:1505] Processing CreateTablet for tablet afd51f330eb649b998a22a343d993b30 (DEFAULT_TABLE table=default.Table_cae5b2188af54d068db115a0745ee866 [id=f4157d7b9fc848a8b7fa08742b456167]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.138171 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd51f330eb649b998a22a343d993b30. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.138455 1017 tablet_service.cc:1505] Processing CreateTablet for tablet bd8226b431c6456885e928880938969d (DEFAULT_TABLE table=default.Table_3900abc2f8dc49ceb70dc23666566d3f [id=6fba2831971a4bffac786611b54a6468]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.138844 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd8226b431c6456885e928880938969d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.129016 888 tablet_service.cc:1505] Processing CreateTablet for tablet 2b92b6d5dde2465684d19aee44a83160 (DEFAULT_TABLE table=default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.139592 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b92b6d5dde2465684d19aee44a83160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.140069 1022 tablet_service.cc:1505] Processing CreateTablet for tablet fe4086fc7a4f4db0b7541f06c9833aab (DEFAULT_TABLE table=default.Table_8076bc8b31ba411faf089f314e6ed2ef [id=25016f04bb864f92a1cdfb6ceaa9fba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.140595 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe4086fc7a4f4db0b7541f06c9833aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.141564 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 (DEFAULT_TABLE table=default.Table_69c0719fe94a4c7594516a739f6a40ea [id=a5424dd4eee64d0686d5e4a8a5c35213]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.141943 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b86f4adaf4540f48a4f42bc45e259e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.143508 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 88b9ca42a5ce4572aebe2a24f4758b12 (DEFAULT_TABLE table=default.Table_5242f3b282284ab7aff8525106732d53 [id=0d173bfba2b343dbac6a02a5ff84c828]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.143874 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b9ca42a5ce4572aebe2a24f4758b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.128767 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_c16ccd85e43049d69fa14c92e073791c" } new_table_name: "default.Table_a30f394f4b6e4b3385cb6abafe21b8aa" modify_external_catalogs: true
I20251214 20:52:12.130363 517 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8140dad1fab142d98b930b53ae5a64c2 (table default.Table_d6e5176452664864914a6f76f32af5ac [id=3606c9fa9a3d4b55b86dddf293442a3c]). Expected version 1 got 0
W20251214 20:52:12.149828 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 9523d72fd91445539d67e1eac2a80afd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9523d72fd91445539d67e1eac2a80afd already in progress: creating tablet
W20251214 20:52:12.150143 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ed1d007f1ca24b7291262708ddfca3c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed1d007f1ca24b7291262708ddfca3c3 already in progress: creating tablet
W20251214 20:52:12.150403 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f3a2143b5508445492b970d2588cf1cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3a2143b5508445492b970d2588cf1cf already in progress: creating tablet
I20251214 20:52:12.152835 1202 tablet_bootstrap.cc:654] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:12.153752 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c16895ca08d94b56b5c4bc9352e330ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c16895ca08d94b56b5c4bc9352e330ef already in progress: creating tablet
I20251214 20:52:12.153999 886 tablet_service.cc:1505] Processing CreateTablet for tablet 52daf1204bf142e7b82852bdec6582fb (DEFAULT_TABLE table=default.Table_bff6195ce6594ee9b4a55eb6058dcb63 [id=a0e13ae34009499d91b4461520c292c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.154600 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52daf1204bf142e7b82852bdec6582fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.159839 876 tablet_service.cc:1505] Processing CreateTablet for tablet bd8226b431c6456885e928880938969d (DEFAULT_TABLE table=default.Table_3900abc2f8dc49ceb70dc23666566d3f [id=6fba2831971a4bffac786611b54a6468]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.160329 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd8226b431c6456885e928880938969d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.160408 1208 ts_tablet_manager.cc:1929] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.160578 1208 log.cc:1199] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/2a0eb9ab79d9428997c6f66790d6b1a3
I20251214 20:52:12.160979 1208 ts_tablet_manager.cc:1950] T 2a0eb9ab79d9428997c6f66790d6b1a3 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:12.161453 878 tablet_service.cc:1505] Processing CreateTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 (DEFAULT_TABLE table=default.Table_69c0719fe94a4c7594516a739f6a40ea [id=a5424dd4eee64d0686d5e4a8a5c35213]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.161499 1208 tablet_replica.cc:333] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:12.162247 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 2a0eb9ab79d9428997c6f66790d6b1a3 (table default.Table_efa9fa90866441c8a4d8a14d14bb4bdd [id=0a848dd5f6474712b2eba70add3743dc]) successfully deleted
I20251214 20:52:12.162725 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b86f4adaf4540f48a4f42bc45e259e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.127007 748 tablet_service.cc:1505] Processing CreateTablet for tablet 88b9ca42a5ce4572aebe2a24f4758b12 (DEFAULT_TABLE table=default.Table_5242f3b282284ab7aff8525106732d53 [id=0d173bfba2b343dbac6a02a5ff84c828]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.165906 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b9ca42a5ce4572aebe2a24f4758b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.127461 745 tablet_service.cc:1505] Processing CreateTablet for tablet afd51f330eb649b998a22a343d993b30 (DEFAULT_TABLE table=default.Table_cae5b2188af54d068db115a0745ee866 [id=f4157d7b9fc848a8b7fa08742b456167]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.168030 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd51f330eb649b998a22a343d993b30. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:12.168264 518 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c25832da5c1644329e18653000f3330e consensus state has no leader
I20251214 20:52:12.168478 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_0fd459d9720046cc9ff44aaf767d2994"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:12.168661 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_1fe00d64c7144563a94dac2fb9a0b29f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.169972 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fe00d64c7144563a94dac2fb9a0b29f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.127820 749 tablet_service.cc:1505] Processing CreateTablet for tablet 2b92b6d5dde2465684d19aee44a83160 (DEFAULT_TABLE table=default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.171474 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b92b6d5dde2465684d19aee44a83160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.171730 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_8e3f7e1435ea493fa4729ca0d9e23573" } modify_external_catalogs: true
I20251214 20:52:12.171816 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_031ef16eb9c246b6ab4ce3f1661376de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:12.171926 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_8e3f7e1435ea493fa4729ca0d9e23573" } modify_external_catalogs: true
W20251214 20:52:12.172345 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_031ef16eb9c246b6ab4ce3f1661376de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.173004 1415 raft_consensus.cc:493] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.173200 1415 raft_consensus.cc:515] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:12.169972 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fd459d9720046cc9ff44aaf767d2994 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.174199 1415 leader_election.cc:290] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.174855 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f44e7223f79041ec9d66d0a1bdcd4ff5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.128153 756 tablet_service.cc:1505] Processing CreateTablet for tablet fe4086fc7a4f4db0b7541f06c9833aab (DEFAULT_TABLE table=default.Table_8076bc8b31ba411faf089f314e6ed2ef [id=25016f04bb864f92a1cdfb6ceaa9fba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.131578 742 tablet_service.cc:1505] Processing CreateTablet for tablet 52daf1204bf142e7b82852bdec6582fb (DEFAULT_TABLE table=default.Table_bff6195ce6594ee9b4a55eb6058dcb63 [id=a0e13ae34009499d91b4461520c292c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.175096 1055 raft_consensus.cc:2468] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:12.175423 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52daf1204bf142e7b82852bdec6582fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.175681 713 leader_election.cc:304] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:12.176017 1341 raft_consensus.cc:2804] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:12.176131 1341 raft_consensus.cc:493] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:12.176206 1341 raft_consensus.cc:3060] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.132625 755 tablet_service.cc:1505] Processing CreateTablet for tablet bd8226b431c6456885e928880938969d (DEFAULT_TABLE table=default.Table_3900abc2f8dc49ceb70dc23666566d3f [id=6fba2831971a4bffac786611b54a6468]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.133406 751 tablet_service.cc:1505] Processing CreateTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 (DEFAULT_TABLE table=default.Table_69c0719fe94a4c7594516a739f6a40ea [id=a5424dd4eee64d0686d5e4a8a5c35213]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.154709 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ba2d34a6d2645bd97fe85fed952692b with delete_type TABLET_DATA_DELETED (replaced by tablet f0d661a1bb034053a4634ff537eafc7d) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.154894 754 tablet_service.cc:1552] Processing DeleteTablet for tablet d8947ebdc87341ad8b1a01f5dd246172 with delete_type TABLET_DATA_DELETED (replaced by tablet d13f93bb5e5f4b59bcf9f4bfed070f25) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.176741 1206 ts_tablet_manager.cc:1434] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.054s user 0.004s sys 0.000s
I20251214 20:52:12.176836 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe4086fc7a4f4db0b7541f06c9833aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.177037 1206 tablet_bootstrap.cc:492] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:12.179154 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b86f4adaf4540f48a4f42bc45e259e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.155063 1207 ts_tablet_manager.cc:1929] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:12.187392 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd8226b431c6456885e928880938969d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.139168 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 2b92b6d5dde2465684d19aee44a83160 (DEFAULT_TABLE table=default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.191210 1207 log.cc:1199] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/6cc54be0bc6d4f259967fbe0a5023f82
I20251214 20:52:12.191267 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.192047 1207 ts_tablet_manager.cc:1950] T 6cc54be0bc6d4f259967fbe0a5023f82 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:12.192444 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 115) took 1213 ms. Trace:
I20251214 20:52:12.192544 1207 rpcz_store.cc:276] 1214 20:52:10.979066 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.979174 (+ 108us) service_pool.cc:225] Handling call
1214 20:52:12.105144 (+1125970us) ts_tablet_manager.cc:1151] Deleting tablet 6cc54be0bc6d4f259967fbe0a5023f82
1214 20:52:12.105158 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.111850 (+ 6692us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.154893 (+ 43043us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.192378 (+ 37485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123425}
I20251214 20:52:12.191316 1206 tablet_bootstrap.cc:654] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.193023 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 6cc54be0bc6d4f259967fbe0a5023f82 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]) successfully deleted
I20251214 20:52:12.196452 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 116) took 1217 ms. Trace:
I20251214 20:52:12.196549 1207 rpcz_store.cc:276] 1214 20:52:10.979209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.979296 (+ 87us) service_pool.cc:225] Handling call
1214 20:52:12.196241 (+1216945us) ts_tablet_manager.cc:1151] Deleting tablet 8e2ee5cc95304fc1a80ed4d31838ac38
1214 20:52:12.196252 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.196395 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213103}
W20251214 20:52:12.197021 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e2ee5cc95304fc1a80ed4d31838ac38 already in progress: creating tablet
I20251214 20:52:12.197484 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 114) took 1218 ms. Trace:
I20251214 20:52:12.197573 1207 rpcz_store.cc:276] 1214 20:52:10.978905 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.981155 (+ 2250us) service_pool.cc:225] Handling call
1214 20:52:12.197300 (+1216145us) ts_tablet_manager.cc:1151] Deleting tablet 63497e77baeb4d328be2421b5732a28a
1214 20:52:12.197309 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.197434 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213549}
W20251214 20:52:12.197990 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 63497e77baeb4d328be2421b5732a28a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63497e77baeb4d328be2421b5732a28a already in progress: creating tablet
I20251214 20:52:12.198163 1208 raft_consensus.cc:2243] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.191092 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b92b6d5dde2465684d19aee44a83160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.201702 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 6bd6856b800247de896f16eee66f2827
I20251214 20:52:12.201776 1213 raft_consensus.cc:493] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.202225 524 catalog_manager.cc:6724] ReplaceTablet: tablet 6bd6856b800247de896f16eee66f2827 deleted and replaced by tablet 3c3962c33a294fa98f8bd485268aee71
I20251214 20:52:12.202355 1213 raft_consensus.cc:515] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.203213 1101 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:12.203459 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.200155 1203 tablet_bootstrap.cc:654] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.204835 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f44e7223f79041ec9d66d0a1bdcd4ff5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:12.205367 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet b1df0dad2fb040a5a12e0401e00c87db
I20251214 20:52:12.205651 915 raft_consensus.cc:2468] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:12.205837 520 catalog_manager.cc:6724] ReplaceTablet: tablet b1df0dad2fb040a5a12e0401e00c87db deleted and replaced by tablet 8fcdd6bafa5c4aa6b59668cde523d010
I20251214 20:52:12.206044 520 master_service.cc:954] ReplaceTablet: received request to replace tablet ba50e32f7e724868bb891045a939e362 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.206446 523 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 44f51e11041841e98b9a15d380226e4e
I20251214 20:52:12.206875 523 catalog_manager.cc:6724] ReplaceTablet: tablet 44f51e11041841e98b9a15d380226e4e deleted and replaced by tablet 5f4760585baf4b4ab3b9b94b5a748a61
I20251214 20:52:12.208078 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_e8acf2f630434ce3b7f06a4a01e4a2a3" } modify_external_catalogs: true
I20251214 20:52:12.208284 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_e8acf2f630434ce3b7f06a4a01e4a2a3" } modify_external_catalogs: true
I20251214 20:52:12.209811 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 358d5c6b594147c8ba136b4b0732fae1
I20251214 20:52:12.210273 522 catalog_manager.cc:6724] ReplaceTablet: tablet 358d5c6b594147c8ba136b4b0732fae1 deleted and replaced by tablet 3bbb28259c1442ae94f4eff76c25c539
I20251214 20:52:12.210572 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_01dcfe83d7854b4e9b8d6f62a557a0fe" } modify_external_catalogs: true
I20251214 20:52:12.210745 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_01dcfe83d7854b4e9b8d6f62a557a0fe" } modify_external_catalogs: true
I20251214 20:52:12.212368 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ca875b1006554ba6845c0cf23671a8da" } modify_external_catalogs: true
I20251214 20:52:12.212553 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ca875b1006554ba6845c0cf23671a8da" } modify_external_catalogs: true
I20251214 20:52:12.213940 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd6856b800247de896f16eee66f2827 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c3962c33a294fa98f8bd485268aee71) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.215024 1213 leader_election.cc:290] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.217171 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2dab958dcddb48e28bfc246163f539ae" } modify_external_catalogs: true
I20251214 20:52:12.217368 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2dab958dcddb48e28bfc246163f539ae" } modify_external_catalogs: true
I20251214 20:52:12.220911 754 tablet_service.cc:1552] Processing DeleteTablet for tablet b1df0dad2fb040a5a12e0401e00c87db with delete_type TABLET_DATA_DELETED (replaced by tablet 8fcdd6bafa5c4aa6b59668cde523d010) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.221238 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 44f51e11041841e98b9a15d380226e4e with delete_type TABLET_DATA_DELETED (replaced by tablet 5f4760585baf4b4ab3b9b94b5a748a61) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.221421 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 358d5c6b594147c8ba136b4b0732fae1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bbb28259c1442ae94f4eff76c25c539) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.221660 754 tablet_service.cc:1552] Processing DeleteTablet for tablet f2b66900d6ea42aa849c24acbbe23c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.222056 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd6856b800247de896f16eee66f2827 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c3962c33a294fa98f8bd485268aee71) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.222445 889 tablet_service.cc:1552] Processing DeleteTablet for tablet b1df0dad2fb040a5a12e0401e00c87db with delete_type TABLET_DATA_DELETED (replaced by tablet 8fcdd6bafa5c4aa6b59668cde523d010) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.222545 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 113) took 1272 ms. Trace:
I20251214 20:52:12.224568 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 44f51e11041841e98b9a15d380226e4e with delete_type TABLET_DATA_DELETED (replaced by tablet 5f4760585baf4b4ab3b9b94b5a748a61) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.225584 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "119c8cc0d6eb4d7f9a5d7fb3000c347b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:12.226928 848 leader_election.cc:343] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.224195 1207 rpcz_store.cc:276] 1214 20:52:10.950021 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.950163 (+ 142us) service_pool.cc:225] Handling call
1214 20:52:12.222325 (+1272162us) ts_tablet_manager.cc:1151] Deleting tablet 6b4f582d8165439d86d9228c146dc41a
1214 20:52:12.222339 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.222487 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237277}
I20251214 20:52:12.227556 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 358d5c6b594147c8ba136b4b0732fae1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bbb28259c1442ae94f4eff76c25c539) from {username='slave'} at 127.0.0.1:43856
W20251214 20:52:12.230324 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 6b4f582d8165439d86d9228c146dc41a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b4f582d8165439d86d9228c146dc41a already in progress: creating tablet
I20251214 20:52:12.230963 1207 tablet_replica.cc:333] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:12.231211 1207 raft_consensus.cc:2243] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.231441 1207 raft_consensus.cc:2272] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.231853 1341 raft_consensus.cc:515] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.234884 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f44e7223f79041ec9d66d0a1bdcd4ff5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:12.235275 915 raft_consensus.cc:3060] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.235512 1207 ts_tablet_manager.cc:1916] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.237725 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd6856b800247de896f16eee66f2827 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c3962c33a294fa98f8bd485268aee71) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.238361 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet b1df0dad2fb040a5a12e0401e00c87db with delete_type TABLET_DATA_DELETED (replaced by tablet 8fcdd6bafa5c4aa6b59668cde523d010) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.241240 915 raft_consensus.cc:2468] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:12.245234 1342 raft_consensus.cc:493] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.245460 1342 raft_consensus.cc:515] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.245812 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 358d5c6b594147c8ba136b4b0732fae1 with delete_type TABLET_DATA_DELETED (replaced by tablet 3bbb28259c1442ae94f4eff76c25c539) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.246284 1342 leader_election.cc:290] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.246472 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 44f51e11041841e98b9a15d380226e4e with delete_type TABLET_DATA_DELETED (replaced by tablet 5f4760585baf4b4ab3b9b94b5a748a61) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:12.246825 525 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 144dee1a6d7344f2b79da6e79e67599e consensus state has no leader
I20251214 20:52:12.252296 1208 raft_consensus.cc:2272] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.257405 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "119c8cc0d6eb4d7f9a5d7fb3000c347b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.257926 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47a67b1b9dc94e498cad91a72e655a75" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.258538 1205 tablet_replica.cc:333] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:12.259569 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 6bd6856b800247de896f16eee66f2827 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bd6856b800247de896f16eee66f2827 already in progress: creating tablet
W20251214 20:52:12.259836 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b1df0dad2fb040a5a12e0401e00c87db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1df0dad2fb040a5a12e0401e00c87db already in progress: creating tablet
I20251214 20:52:12.260599 1205 raft_consensus.cc:2243] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.260916 1205 raft_consensus.cc:2272] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:12.261708 851 leader_election.cc:343] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.261905 851 leader_election.cc:304] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:12.262195 1342 raft_consensus.cc:2749] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:12.262398 851 leader_election.cc:343] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.263504 712 leader_election.cc:304] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:12.263734 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.264012 1415 raft_consensus.cc:2804] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:12.264712 1415 raft_consensus.cc:697] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:12.265594 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47a67b1b9dc94e498cad91a72e655a75" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:12.265771 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f7aff79ce154385a73c37b10b9c2430 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.265929 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
W20251214 20:52:12.266759 848 leader_election.cc:343] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: Tablet not RUNNING: STOPPED
I20251214 20:52:12.266927 848 leader_election.cc:304] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:12.267290 1342 raft_consensus.cc:2749] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:12.265072 1415 consensus_queue.cc:237] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.270884 1205 ts_tablet_manager.cc:1916] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.273448 1207 ts_tablet_manager.cc:1929] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.273628 1207 log.cc:1199] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/47a67b1b9dc94e498cad91a72e655a75
I20251214 20:52:12.275218 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.275446 1207 ts_tablet_manager.cc:1950] T 47a67b1b9dc94e498cad91a72e655a75 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:12.275807 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 118) took 1277 ms. Trace:
I20251214 20:52:12.276086 1207 rpcz_store.cc:276] 1214 20:52:10.999044 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:10.999375 (+ 331us) service_pool.cc:225] Handling call
1214 20:52:12.230851 (+1231476us) ts_tablet_manager.cc:1151] Deleting tablet 47a67b1b9dc94e498cad91a72e655a75
1214 20:52:12.230864 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.241591 (+ 10727us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.273025 (+ 31434us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.275776 (+ 2751us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1227809}
I20251214 20:52:12.276465 1207 tablet_replica.cc:333] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:12.278851 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 47a67b1b9dc94e498cad91a72e655a75 (table default.Table_fa0e830600ba474bbbd6a68b9593d5f2 [id=16f2e0f0dbda4548871716a4a0f44dd4]) successfully deleted
I20251214 20:52:12.279862 1341 leader_election.cc:290] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.281807 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f44e7223f79041ec9d66d0a1bdcd4ff5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:12.282044 1202 tablet_bootstrap.cc:492] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:12.282054 1056 raft_consensus.cc:3060] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.285115 1056 raft_consensus.cc:2468] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:12.279949 1207 raft_consensus.cc:2243] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.286312 1207 raft_consensus.cc:2272] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.287442 1207 ts_tablet_manager.cc:1916] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.292198 1202 ts_tablet_manager.cc:1403] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.163s user 0.004s sys 0.000s
I20251214 20:52:12.292541 1208 ts_tablet_manager.cc:1916] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.293337 1202 raft_consensus.cc:359] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.293578 1202 raft_consensus.cc:385] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.293653 1202 raft_consensus.cc:740] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.293983 1202 consensus_queue.cc:260] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.293972 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_d863e40d2a9e46649f9dc97f33a616db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.294548 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d863e40d2a9e46649f9dc97f33a616db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.296027 1202 ts_tablet_manager.cc:1434] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:12.296413 1202 tablet_bootstrap.cc:492] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:12.298003 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 144dee1a6d7344f2b79da6e79e67599e (table default.Table_a30f394f4b6e4b3385cb6abafe21b8aa [id=8cd62da3d57c4be4b5f10ace49d2e148], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 144dee1a6d7344f2b79da6e79e67599e consensus state has no leader
I20251214 20:52:12.301587 1205 ts_tablet_manager.cc:1929] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.302755 1205 log.cc:1199] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/358d5c6b594147c8ba136b4b0732fae1
I20251214 20:52:12.304569 521 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet ba50e32f7e724868bb891045a939e362
I20251214 20:52:12.305156 1205 ts_tablet_manager.cc:1950] T 358d5c6b594147c8ba136b4b0732fae1 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:12.305317 1206 tablet_bootstrap.cc:492] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:12.305380 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_997191dc162f47a4b9e3b05289f6cb48" } new_table_name: "default.Table_6d9954b482a4427ba8a996c085f831bf" modify_external_catalogs: true
I20251214 20:52:12.305501 1206 ts_tablet_manager.cc:1403] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.129s user 0.024s sys 0.024s
I20251214 20:52:12.306284 1206 raft_consensus.cc:359] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.306493 1206 raft_consensus.cc:385] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.306577 1206 raft_consensus.cc:740] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.307005 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_0071d629c0444fa89511b0938f2e757e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.307636 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0071d629c0444fa89511b0938f2e757e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.311439 1205 tablet_replica.cc:333] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:12.312639 1205 raft_consensus.cc:2243] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:12.314154 876 tablet_service.cc:1552] Processing DeleteTablet for tablet ba50e32f7e724868bb891045a939e362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.315434 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 358d5c6b594147c8ba136b4b0732fae1 (table default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]) successfully deleted
I20251214 20:52:12.315935 1206 consensus_queue.cc:260] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.316684 1206 ts_tablet_manager.cc:1434] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20251214 20:52:12.316666 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet ba50e32f7e724868bb891045a939e362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.316989 1202 tablet_bootstrap.cc:654] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.317063 1206 tablet_bootstrap.cc:492] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:12.316259 1205 raft_consensus.cc:2272] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.319267 1203 tablet_bootstrap.cc:492] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:12.319463 1203 ts_tablet_manager.cc:1403] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.209s user 0.005s sys 0.009s
I20251214 20:52:12.320946 1203 raft_consensus.cc:359] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.321209 1203 raft_consensus.cc:385] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.321297 1203 raft_consensus.cc:740] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.321810 1203 consensus_queue.cc:260] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.322779 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.321694 1205 ts_tablet_manager.cc:1916] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.324556 1203 ts_tablet_manager.cc:1434] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251214 20:52:12.327565 742 tablet_service.cc:1552] Processing DeleteTablet for tablet ba50e32f7e724868bb891045a939e362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.327303 1203 tablet_bootstrap.cc:492] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:12.328388 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.329733 1206 tablet_bootstrap.cc:654] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.331934 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.336229 1342 raft_consensus.cc:493] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.336510 1342 raft_consensus.cc:515] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.337396 1342 leader_election.cc:290] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.339268 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3b9ff3c6234507b76afd18f097b6e0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:12.339951 848 leader_election.cc:343] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.340752 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.343369 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet f44e7223f79041ec9d66d0a1bdcd4ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.343612 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c3b9ff3c6234507b76afd18f097b6e0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:12.344549 851 leader_election.cc:343] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.344713 851 leader_election.cc:304] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:12.345193 1342 raft_consensus.cc:2749] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:12.346844 1208 ts_tablet_manager.cc:1929] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.348100 1206 tablet_bootstrap.cc:492] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:12.348287 1206 ts_tablet_manager.cc:1403] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.003s
I20251214 20:52:12.348119 1203 tablet_bootstrap.cc:654] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.348258 1208 log.cc:1199] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/0a0ca65d369a45958c6dac539964114b
I20251214 20:52:12.349181 1206 raft_consensus.cc:359] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.349390 1206 raft_consensus.cc:385] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.349494 1206 raft_consensus.cc:740] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.349771 1206 consensus_queue.cc:260] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.350452 1206 ts_tablet_manager.cc:1434] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:12.350797 1206 tablet_bootstrap.cc:492] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:12.352299 1208 ts_tablet_manager.cc:1950] T 0a0ca65d369a45958c6dac539964114b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:12.353796 1202 tablet_bootstrap.cc:492] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:12.353825 1206 tablet_bootstrap.cc:654] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.353963 1202 ts_tablet_manager.cc:1403] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.002s
I20251214 20:52:12.354903 1202 raft_consensus.cc:359] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.355094 1202 raft_consensus.cc:385] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.355207 1202 raft_consensus.cc:740] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.355646 1207 ts_tablet_manager.cc:1929] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.355487 1202 consensus_queue.cc:260] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.357331 1207 log.cc:1199] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/7bfb575249b8403991d21c6fc96eefeb
I20251214 20:52:12.357443 1202 ts_tablet_manager.cc:1434] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251214 20:52:12.357831 1202 tablet_bootstrap.cc:492] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:12.359802 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 123) took 1117 ms. Trace:
I20251214 20:52:12.365890 1207 ts_tablet_manager.cc:1950] T 7bfb575249b8403991d21c6fc96eefeb P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:12.366686 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_9447e701a9d0498f9c179a48dfb633b5" } new_table_name: "default.Table_cd29381b062249718fbdc39a105f46a7" modify_external_catalogs: true
I20251214 20:52:12.366864 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_db08ad4da8954cba8aebd4c00ee50cd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:12.367203 1207 tablet_replica.cc:333] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:12.367511 1207 raft_consensus.cc:2243] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:12.367974 1207 pending_rounds.cc:70] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
W20251214 20:52:12.367941 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db08ad4da8954cba8aebd4c00ee50cd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.368208 1207 pending_rounds.cc:77] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493700174991360 op_type: NO_OP noop_request { }
I20251214 20:52:12.368315 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 376e5efede87410eb9f95f36bedebf9d
W20251214 20:52:12.368425 521 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d4626d02ca548f3bf5a1a9142600bff consensus state has no leader
I20251214 20:52:12.368443 1207 raft_consensus.cc:2889] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:12.368541 1207 raft_consensus.cc:2272] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.369603 1207 ts_tablet_manager.cc:1916] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.370781 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_fbb5b21626c142c8baac60a7f29d3c51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.371541 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbb5b21626c142c8baac60a7f29d3c51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.371981 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.360618 1208 rpcz_store.cc:276] 1214 20:52:11.241776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:11.241837 (+ 61us) service_pool.cc:225] Handling call
1214 20:52:12.161420 (+919583us) ts_tablet_manager.cc:1151] Deleting tablet 0a0ca65d369a45958c6dac539964114b
1214 20:52:12.161432 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.316353 (+154921us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.344974 (+ 28621us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.359416 (+ 14442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":908412}
I20251214 20:52:12.377944 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 60ae3db968f6412aa0c3952fafe65f88
I20251214 20:52:12.378366 1342 raft_consensus.cc:493] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.378593 1342 raft_consensus.cc:515] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.378708 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 0a0ca65d369a45958c6dac539964114b (table default.Table_28368c3015e84594800ff8a110b0b4c3 [id=45ca58c541924d98bf3a7f04078ac204]) successfully deleted
I20251214 20:52:12.379391 1342 leader_election.cc:290] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.379592 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet d768c45bc02b4d108798efbf3f6aaf7d
I20251214 20:52:12.379683 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.365435 1397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=17 ignored=0} inserts{seen=23 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251214 20:52:12.380074 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a05f1ec28fb24508ae937c70aaaefffe" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.380322 1056 raft_consensus.cc:2468] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:12.380797 1397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: Bootstrap complete.
W20251214 20:52:12.380769 848 leader_election.cc:343] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: a05f1ec28fb24508ae937c70aaaefffe
I20251214 20:52:12.380896 848 leader_election.cc:304] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:12.381537 878 tablet_service.cc:1552] Processing DeleteTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.381765 1342 raft_consensus.cc:2804] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:12.381856 1342 raft_consensus.cc:493] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:12.381958 1342 raft_consensus.cc:3060] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.382591 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 376e5efede87410eb9f95f36bedebf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.384922 1207 ts_tablet_manager.cc:1929] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.385087 1207 log.cc:1199] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/f44e7223f79041ec9d66d0a1bdcd4ff5
I20251214 20:52:12.385325 1342 raft_consensus.cc:515] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.385546 1207 ts_tablet_manager.cc:1950] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:12.386683 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_efa9fa90866441c8a4d8a14d14bb4bdd" } new_table_name: "default.Table_0870823e1f29409f9d61ebee0bbd060d" modify_external_catalogs: true
W20251214 20:52:12.387313 848 leader_election.cc:343] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: a05f1ec28fb24508ae937c70aaaefffe
I20251214 20:52:12.387845 1342 leader_election.cc:290] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.388302 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a05f1ec28fb24508ae937c70aaaefffe" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:12.388656 1056 raft_consensus.cc:3060] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.388453 1208 tablet_replica.cc:333] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:12.390412 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_ab4c9e0bdaf74a1791ddad541ece2482"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.391155 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab4c9e0bdaf74a1791ddad541ece2482 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.390709 1208 raft_consensus.cc:2243] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:12.392004 1056 raft_consensus.cc:2468] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:12.392644 851 leader_election.cc:304] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:12.392915 1203 tablet_bootstrap.cc:492] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:12.392802 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3385c79468e4468eb0cddb80a5cf0c75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:12.393189 1203 ts_tablet_manager.cc:1403] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.004s
I20251214 20:52:12.393548 1342 raft_consensus.cc:2804] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:12.393730 1342 raft_consensus.cc:697] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:12.393795 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet a05f1ec28fb24508ae937c70aaaefffe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.394024 1342 consensus_queue.cc:237] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.394671 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.394968 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 376e5efede87410eb9f95f36bedebf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.395092 1203 raft_consensus.cc:359] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.395483 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.395500 1203 raft_consensus.cc:385] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.395787 1203 raft_consensus.cc:740] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.396119 1203 consensus_queue.cc:260] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:12.396479 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3385c79468e4468eb0cddb80a5cf0c75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.396664 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.396692 1205 ts_tablet_manager.cc:1929] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.397469 1205 log.cc:1199] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/44f51e11041841e98b9a15d380226e4e
I20251214 20:52:12.397985 1203 ts_tablet_manager.cc:1434] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251214 20:52:12.398561 1203 tablet_bootstrap.cc:492] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:12.398521 1208 pending_rounds.cc:70] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:12.400421 1208 pending_rounds.cc:77] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493696450203648 op_type: NO_OP noop_request { }
I20251214 20:52:12.401422 1205 ts_tablet_manager.cc:1950] T 44f51e11041841e98b9a15d380226e4e P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:12.401924 1203 tablet_bootstrap.cc:654] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.401589 1208 raft_consensus.cc:2889] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:12.402684 1208 raft_consensus.cc:2272] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:12.403738 510 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42865, user_credentials={real_user=slave}} blocked reactor thread for 43490.3us
I20251214 20:52:12.404404 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 376e5efede87410eb9f95f36bedebf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.405104 742 tablet_service.cc:1552] Processing DeleteTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.405449 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 7bfb575249b8403991d21c6fc96eefeb (table default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3]) successfully deleted
W20251214 20:52:12.405870 510 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 0a0ca65d369a45958c6dac539964114b because the tablet was not found. No further retry: Not found: Tablet not found: 0a0ca65d369a45958c6dac539964114b
I20251214 20:52:12.406113 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet f44e7223f79041ec9d66d0a1bdcd4ff5 (table default.Table_eb5f20f2b40e458cade7599dbd9d5aa7 [id=e5f2cef45cbf4a69977c9be3799f9330]) successfully deleted
I20251214 20:52:12.406145 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
W20251214 20:52:12.406320 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2842f1dc0c374d24946f90dac1cf8aa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2842f1dc0c374d24946f90dac1cf8aa5 already in progress: creating tablet
W20251214 20:52:12.406541 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 9523d72fd91445539d67e1eac2a80afd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9523d72fd91445539d67e1eac2a80afd already in progress: creating tablet
W20251214 20:52:12.406780 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c16895ca08d94b56b5c4bc9352e330ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c16895ca08d94b56b5c4bc9352e330ef already in progress: creating tablet
I20251214 20:52:12.406805 1207 tablet_replica.cc:333] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
W20251214 20:52:12.406994 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ed1d007f1ca24b7291262708ddfca3c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed1d007f1ca24b7291262708ddfca3c3 already in progress: creating tablet
W20251214 20:52:12.407235 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f3a2143b5508445492b970d2588cf1cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3a2143b5508445492b970d2588cf1cf already in progress: creating tablet
I20251214 20:52:12.407829 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 44f51e11041841e98b9a15d380226e4e (table default.Table_0191351f0cd945e489dc8d02dbe2c5be [id=3aaa67725e8442fdbae975274c8596c4]) successfully deleted
I20251214 20:52:12.409588 1205 tablet_replica.cc:333] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:12.407303 1207 raft_consensus.cc:2243] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.410847 1207 raft_consensus.cc:2272] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.410698 1205 raft_consensus.cc:2243] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.411569 1205 raft_consensus.cc:2272] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.413178 1208 ts_tablet_manager.cc:1916] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.414171 1205 ts_tablet_manager.cc:1916] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.414620 1207 ts_tablet_manager.cc:1916] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:12.435364 517 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 956e4b4493f24109ab1251dde0a4cfc0 consensus state has no leader
I20251214 20:52:12.437341 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_6c56ebc90d1a4b92bc80b11ed334a791" } new_table_name: "default.Table_f93eb4d354254ec79674c615721834b7" modify_external_catalogs: true
I20251214 20:52:12.437443 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_bb6020a9d77b42b5955933a449edc238"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:12.437850 1206 tablet_bootstrap.cc:492] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:12.438032 1206 ts_tablet_manager.cc:1403] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.004s
W20251214 20:52:12.438359 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb6020a9d77b42b5955933a449edc238 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.438930 1206 raft_consensus.cc:359] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.439200 1206 raft_consensus.cc:385] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.439282 1206 raft_consensus.cc:740] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.439421 524 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:48854 (request call id 2) took 1419 ms. Trace:
I20251214 20:52:12.439677 1206 consensus_queue.cc:260] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.439563 524 rpcz_store.cc:276] 1214 20:52:11.020222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.439039 (+1418817us) service_pool.cc:225] Handling call
1214 20:52:12.439350 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251214 20:52:12.440601 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a142157acbcd44adb6099b4428103ee5 consensus state has no leader
I20251214 20:52:12.440830 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 5f7aff79ce154385a73c37b10b9c2430 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.442377 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 2842f1dc0c374d24946f90dac1cf8aa5 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.442909 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 5ba2d34a6d2645bd97fe85fed952692b from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.444334 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c3b9ff3c6234507b76afd18f097b6e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 649f5bd67cbb4f61b7066269f0bf9266) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.444608 1206 ts_tablet_manager.cc:1434] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251214 20:52:12.448817 1203 tablet_bootstrap.cc:492] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:12.449234 1203 ts_tablet_manager.cc:1403] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.002s
I20251214 20:52:12.449586 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_18dc9de393304efcb3ec4eb6be858624" } new_table_name: "default.Table_828c0037b27e439a9dbb15886bb51344" modify_external_catalogs: true
I20251214 20:52:12.449759 1206 tablet_bootstrap.cc:492] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:12.450824 1203 raft_consensus.cc:359] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.451367 1203 raft_consensus.cc:385] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.451552 1203 raft_consensus.cc:740] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.452047 519 master_service.cc:954] ReplaceTablet: received request to replace tablet c25832da5c1644329e18653000f3330e from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.453075 822 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:12.453333 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.455715 1206 tablet_bootstrap.cc:654] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.454941 1203 consensus_queue.cc:260] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.456830 493 master_runner.cc:186] Error getting master registration for 127.0.104.60:35099: OK, instance_id { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" instance_seqno: 1765745531774202 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251214 20:52:12.457931 493 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251214 20:52:12.460070 1203 ts_tablet_manager.cc:1434] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.010s user 0.002s sys 0.002s
I20251214 20:52:12.463064 1203 tablet_bootstrap.cc:492] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:12.464846 1205 ts_tablet_manager.cc:1929] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.465858 1205 log.cc:1199] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/5f7aff79ce154385a73c37b10b9c2430
I20251214 20:52:12.463194 1101 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:12.467485 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.468621 1205 ts_tablet_manager.cc:1950] T 5f7aff79ce154385a73c37b10b9c2430 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:12.469027 1207 ts_tablet_manager.cc:1929] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.423207 1202 tablet_bootstrap.cc:654] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.471972 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 5f7aff79ce154385a73c37b10b9c2430 (table default.Table_170ec365d121403eaaa13bb81ada4535 [id=4b90d1a1fbf649ce9406db32b838ed5b]) successfully deleted
I20251214 20:52:12.454111 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.465830 1397 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:12.423102 1208 ts_tablet_manager.cc:1929] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.475371 1208 log.cc:1199] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ba2ed363bbab4f1f865b183297c98daa
I20251214 20:52:12.475838 1207 log.cc:1199] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/0c344296be0549e88d4811103eb6fee5
I20251214 20:52:12.475884 1208 ts_tablet_manager.cc:1950] T ba2ed363bbab4f1f865b183297c98daa P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:12.476611 1207 ts_tablet_manager.cc:1950] T 0c344296be0549e88d4811103eb6fee5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:12.476940 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 124) took 1043 ms. Trace:
I20251214 20:52:12.478183 1208 rpcz_store.cc:276] 1214 20:52:11.433108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:11.433301 (+ 193us) service_pool.cc:225] Handling call
1214 20:52:12.378831 (+945530us) ts_tablet_manager.cc:1151] Deleting tablet ba2ed363bbab4f1f865b183297c98daa
1214 20:52:12.378849 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.419157 (+ 40308us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.422934 (+ 3777us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.476876 (+ 53942us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":944541}
W20251214 20:52:12.478335 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ba50e32f7e724868bb891045a939e362 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba50e32f7e724868bb891045a939e362 already in progress: creating tablet
I20251214 20:52:12.470008 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.480557 1205 tablet_replica.cc:333] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:12.480635 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 0c344296be0549e88d4811103eb6fee5 (table default.Table_3c3f2d9e5c2b47ed8038a7402612b093 [id=163e1e9d48bc4f51b308383354454363]) successfully deleted
I20251214 20:52:12.482079 1205 raft_consensus.cc:2243] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.482934 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ba2ed363bbab4f1f865b183297c98daa (table default.Table_ad4bec26fc854d67b2ec37f504851a01 [id=17c89c03be10405f834504b803e594c7]) successfully deleted
I20251214 20:52:12.483512 1208 tablet_replica.cc:333] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:12.483784 1203 tablet_bootstrap.cc:654] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.484072 1208 raft_consensus.cc:2243] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.483688 1205 raft_consensus.cc:2272] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.484242 1206 tablet_bootstrap.cc:492] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:12.484503 1206 ts_tablet_manager.cc:1403] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20251214 20:52:12.484786 1208 raft_consensus.cc:2272] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.485462 1206 raft_consensus.cc:359] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.485726 1206 raft_consensus.cc:385] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.485841 1206 raft_consensus.cc:740] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.486052 1202 tablet_bootstrap.cc:492] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:12.486125 1206 consensus_queue.cc:260] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.486347 1202 ts_tablet_manager.cc:1403] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.129s user 0.003s sys 0.004s
I20251214 20:52:12.487265 1202 raft_consensus.cc:359] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.487488 1202 raft_consensus.cc:385] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.487557 1202 raft_consensus.cc:740] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.487540 521 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 33a7f355b40f472883e3399209ef6762 (table default.Table_1c75ca10deb842669e2fea0f16ecc983 [id=5272550343144d4baaba07e97ebaa187]). Expected version 1 got 0
I20251214 20:52:12.487851 1202 consensus_queue.cc:260] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.488449 1206 ts_tablet_manager.cc:1434] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:12.488854 1206 tablet_bootstrap.cc:492] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:12.488821 522 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8140dad1fab142d98b930b53ae5a64c2 (table default.Table_d6e5176452664864914a6f76f32af5ac [id=3606c9fa9a3d4b55b86dddf293442a3c]). Expected version 1 got 0
I20251214 20:52:12.489615 1205 ts_tablet_manager.cc:1916] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.489974 1202 ts_tablet_manager.cc:1434] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:12.490312 1202 tablet_bootstrap.cc:492] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:12.491633 1030 tablet_service.cc:1505] Processing CreateTablet for tablet a10663bd306d40e7a6375d06d69be22d (DEFAULT_TABLE table=default.Table_6c56ebc90d1a4b92bc80b11ed334a791 [id=e8e0419a222b460ab608def1752925f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.491771 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 8af99e0bd8774a678575847a9ec2e557 (DEFAULT_TABLE table=default.Table_ade67d7e10e04c6fb0cfc134947a559d [id=7986a565ce914875bf3e24c1cc2c288a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.492105 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10663bd306d40e7a6375d06d69be22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.493721 1208 ts_tablet_manager.cc:1916] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.494727 1207 tablet_replica.cc:333] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:12.494966 1207 raft_consensus.cc:2243] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.495178 1207 raft_consensus.cc:2272] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.495595 876 tablet_service.cc:1505] Processing CreateTablet for tablet a10663bd306d40e7a6375d06d69be22d (DEFAULT_TABLE table=default.Table_6c56ebc90d1a4b92bc80b11ed334a791 [id=e8e0419a222b460ab608def1752925f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.495829 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.497371 525 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 8140dad1fab142d98b930b53ae5a64c2 (table default.Table_d6e5176452664864914a6f76f32af5ac [id=3606c9fa9a3d4b55b86dddf293442a3c]). Expected version 1 got 0
I20251214 20:52:12.497721 525 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b (table default.Table_a519bf6d1ab8419784be3772e899869d [id=5cbe2438b4294442a9778bf1f641b60e]). Expected version 2 got 0
I20251214 20:52:12.499253 1207 ts_tablet_manager.cc:1916] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.500516 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af99e0bd8774a678575847a9ec2e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.500659 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 0cafed2861e94cd9b0d10d33864ba45d (DEFAULT_TABLE table=default.Table_ba612520b0be474db6573ce754044c8a [id=1bf28f656cfe496d8accd428d0b238f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.501159 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cafed2861e94cd9b0d10d33864ba45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.505929 1207 ts_tablet_manager.cc:1929] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.506260 1021 tablet_service.cc:1505] Processing CreateTablet for tablet cf04f91e79754569a51d468ab576b0dc (DEFAULT_TABLE table=default.Table_7f018aa7a0aa4ab3895001f6c97c48bd [id=98edd613a48b433c9896ff066362554d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.506781 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf04f91e79754569a51d468ab576b0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.496127 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10663bd306d40e7a6375d06d69be22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.508666 1397 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c48b8eaf30c4e2caa94a51d087406d4, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.509624 1397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:12.509750 886 tablet_service.cc:1505] Processing CreateTablet for tablet 0cafed2861e94cd9b0d10d33864ba45d (DEFAULT_TABLE table=default.Table_ba612520b0be474db6573ce754044c8a [id=1bf28f656cfe496d8accd428d0b238f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.510253 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cafed2861e94cd9b0d10d33864ba45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.496501 878 tablet_service.cc:1505] Processing CreateTablet for tablet 8af99e0bd8774a678575847a9ec2e557 (DEFAULT_TABLE table=default.Table_ade67d7e10e04c6fb0cfc134947a559d [id=7986a565ce914875bf3e24c1cc2c288a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.497588 1202 tablet_bootstrap.cc:654] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.513460 1434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:12.513900 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af99e0bd8774a678575847a9ec2e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.514469 889 tablet_service.cc:1505] Processing CreateTablet for tablet cf04f91e79754569a51d468ab576b0dc (DEFAULT_TABLE table=default.Table_7f018aa7a0aa4ab3895001f6c97c48bd [id=98edd613a48b433c9896ff066362554d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.515353 1035 tablet_service.cc:1505] Processing CreateTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 (DEFAULT_TABLE table=default.Table_4b9e6d9e76414e6da9e6767b3463fa10 [id=976ca3fb3a164669abeff0d8eb137e41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.515870 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ae9dc8ead8403cb7215af8778872f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.506130 1206 tablet_bootstrap.cc:654] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.518220 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.521934 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ba2d34a6d2645bd97fe85fed952692b with delete_type TABLET_DATA_DELETED (replaced by tablet f0d661a1bb034053a4634ff537eafc7d) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.524109 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf04f91e79754569a51d468ab576b0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.527957 525 master_service.cc:954] ReplaceTablet: received request to replace tablet ecb69e385bf34709b5d954de4b660087 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.530839 1207 log.cc:1199] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/5f7aff79ce154385a73c37b10b9c2430
I20251214 20:52:12.531077 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c3b9ff3c6234507b76afd18f097b6e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 649f5bd67cbb4f61b7066269f0bf9266) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.531508 1207 ts_tablet_manager.cc:1950] T 5f7aff79ce154385a73c37b10b9c2430 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:12.531500 888 tablet_service.cc:1552] Processing DeleteTablet for tablet d8947ebdc87341ad8b1a01f5dd246172 with delete_type TABLET_DATA_DELETED (replaced by tablet d13f93bb5e5f4b59bcf9f4bfed070f25) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.531630 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 47a67b1b9dc94e498cad91a72e655a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
W20251214 20:52:12.534005 510 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42865, user_credentials={real_user=slave}} blocked reactor thread for 52722.2us
I20251214 20:52:12.534163 1207 tablet_replica.cc:333] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:12.535414 742 tablet_service.cc:1505] Processing CreateTablet for tablet a10663bd306d40e7a6375d06d69be22d (DEFAULT_TABLE table=default.Table_6c56ebc90d1a4b92bc80b11ed334a791 [id=e8e0419a222b460ab608def1752925f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.535822 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a10663bd306d40e7a6375d06d69be22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.537968 755 tablet_service.cc:1505] Processing CreateTablet for tablet 8af99e0bd8774a678575847a9ec2e557 (DEFAULT_TABLE table=default.Table_ade67d7e10e04c6fb0cfc134947a559d [id=7986a565ce914875bf3e24c1cc2c288a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.538332 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af99e0bd8774a678575847a9ec2e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.538683 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.538362 880 tablet_service.cc:1505] Processing CreateTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 (DEFAULT_TABLE table=default.Table_4b9e6d9e76414e6da9e6767b3463fa10 [id=976ca3fb3a164669abeff0d8eb137e41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.539064 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c3b9ff3c6234507b76afd18f097b6e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 649f5bd67cbb4f61b7066269f0bf9266) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.539471 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 5f7aff79ce154385a73c37b10b9c2430 (table default.Table_170ec365d121403eaaa13bb81ada4535 [id=4b90d1a1fbf649ce9406db32b838ed5b]) successfully deleted
I20251214 20:52:12.539744 749 tablet_service.cc:1505] Processing CreateTablet for tablet cf04f91e79754569a51d468ab576b0dc (DEFAULT_TABLE table=default.Table_7f018aa7a0aa4ab3895001f6c97c48bd [id=98edd613a48b433c9896ff066362554d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.540052 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf04f91e79754569a51d468ab576b0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.540077 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f (DEFAULT_TABLE table=default.Table_ec65ee1eda49435895dab50f78b4ee15 [id=8a0182244a254993998d4ea00d03cd9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.540779 745 tablet_service.cc:1505] Processing CreateTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 (DEFAULT_TABLE table=default.Table_4b9e6d9e76414e6da9e6767b3463fa10 [id=976ca3fb3a164669abeff0d8eb137e41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.541082 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ae9dc8ead8403cb7215af8778872f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.541546 888 tablet_service.cc:1505] Processing CreateTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f (DEFAULT_TABLE table=default.Table_ec65ee1eda49435895dab50f78b4ee15 [id=8a0182244a254993998d4ea00d03cd9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.542016 756 tablet_service.cc:1505] Processing CreateTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f (DEFAULT_TABLE table=default.Table_ec65ee1eda49435895dab50f78b4ee15 [id=8a0182244a254993998d4ea00d03cd9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.542341 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c0f49e5d78240a296e47eccb3218e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.544555 882 tablet_service.cc:1505] Processing CreateTablet for tablet 2da66ec83e9140f7a605f0c86abff468 (DEFAULT_TABLE table=default.Table_2325a2c5e97f45ebac4c943380bb3525 [id=0c6d9651f90f479293852cb3fb1d320f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.545269 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 2da66ec83e9140f7a605f0c86abff468 (DEFAULT_TABLE table=default.Table_2325a2c5e97f45ebac4c943380bb3525 [id=0c6d9651f90f479293852cb3fb1d320f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.545805 1022 tablet_service.cc:1505] Processing CreateTablet for tablet c4b22043d81b493fb12116422b1d3fb1 (DEFAULT_TABLE table=default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.546065 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c0f49e5d78240a296e47eccb3218e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.546891 518 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b (table default.Table_a519bf6d1ab8419784be3772e899869d [id=5cbe2438b4294442a9778bf1f641b60e]). Expected version 2 got 0
I20251214 20:52:12.548549 1434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:12.549787 518 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 33a7f355b40f472883e3399209ef6762 (table default.Table_1c75ca10deb842669e2fea0f16ecc983 [id=5272550343144d4baaba07e97ebaa187]). Expected version 1 got 0
I20251214 20:52:12.539506 748 tablet_service.cc:1505] Processing CreateTablet for tablet 0cafed2861e94cd9b0d10d33864ba45d (DEFAULT_TABLE table=default.Table_ba612520b0be474db6573ce754044c8a [id=1bf28f656cfe496d8accd428d0b238f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.550325 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cafed2861e94cd9b0d10d33864ba45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.552296 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2ae9dc8ead8403cb7215af8778872f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.552430 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b22043d81b493fb12116422b1d3fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.552518 889 tablet_service.cc:1505] Processing CreateTablet for tablet c4b22043d81b493fb12116422b1d3fb1 (DEFAULT_TABLE table=default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.553298 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_431bf74dfb32400bb0b2ef098c18e8df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.553972 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_431bf74dfb32400bb0b2ef098c18e8df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.555428 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d6e5176452664864914a6f76f32af5ac" } modify_external_catalogs: true
I20251214 20:52:12.555666 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d6e5176452664864914a6f76f32af5ac" } modify_external_catalogs: true
I20251214 20:52:12.557054 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da66ec83e9140f7a605f0c86abff468. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.570250 751 tablet_service.cc:1505] Processing CreateTablet for tablet c4b22043d81b493fb12116422b1d3fb1 (DEFAULT_TABLE table=default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.570843 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b22043d81b493fb12116422b1d3fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.574117 1205 ts_tablet_manager.cc:1929] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.574271 1205 log.cc:1199] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/f44e7223f79041ec9d66d0a1bdcd4ff5
I20251214 20:52:12.575457 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b22043d81b493fb12116422b1d3fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.578079 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_aa507956e1b54af68730c20f7c10e842"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.578814 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa507956e1b54af68730c20f7c10e842 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.580663 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c0f49e5d78240a296e47eccb3218e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.520548 1397 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: configured and running, proceeding with master startup.
I20251214 20:52:12.529268 1334 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 2 FOLLOWER]: Advancing to term 3
I20251214 20:52:12.583698 886 tablet_service.cc:1552] Processing DeleteTablet for tablet c17126f75b6241c5833269d20774c439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.583987 880 tablet_service.cc:1552] Processing DeleteTablet for tablet a8e09c13caca4a1c9018387972c9f082 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.585180 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_15002dcbc05542cebdf323d46303384b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:12.585750 755 tablet_service.cc:1505] Processing CreateTablet for tablet 2da66ec83e9140f7a605f0c86abff468 (DEFAULT_TABLE table=default.Table_2325a2c5e97f45ebac4c943380bb3525 [id=0c6d9651f90f479293852cb3fb1d320f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.586199 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da66ec83e9140f7a605f0c86abff468. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.587246 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da66ec83e9140f7a605f0c86abff468. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:12.588788 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15002dcbc05542cebdf323d46303384b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.591295 749 tablet_service.cc:1552] Processing DeleteTablet for tablet ebedf28e17a741169e5490822a2d95df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.594553 1205 ts_tablet_manager.cc:1950] T f44e7223f79041ec9d66d0a1bdcd4ff5 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:12.597066 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet f44e7223f79041ec9d66d0a1bdcd4ff5 (table default.Table_eb5f20f2b40e458cade7599dbd9d5aa7 [id=e5f2cef45cbf4a69977c9be3799f9330]) successfully deleted
I20251214 20:52:12.597299 1208 ts_tablet_manager.cc:1929] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.597549 1208 log.cc:1199] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/7bfb575249b8403991d21c6fc96eefeb
I20251214 20:52:12.595307 1203 tablet_bootstrap.cc:492] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:12.598114 1208 ts_tablet_manager.cc:1950] T 7bfb575249b8403991d21c6fc96eefeb P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:12.598296 1203 ts_tablet_manager.cc:1403] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.136s user 0.002s sys 0.005s
I20251214 20:52:12.598642 1205 tablet_replica.cc:333] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:12.599251 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 125) took 1166 ms. Trace:
I20251214 20:52:12.599404 1208 rpcz_store.cc:276] 1214 20:52:11.433234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:11.433644 (+ 410us) service_pool.cc:225] Handling call
1214 20:52:12.483401 (+1049757us) ts_tablet_manager.cc:1151] Deleting tablet 7bfb575249b8403991d21c6fc96eefeb
1214 20:52:12.483414 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.562168 (+ 78754us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.596925 (+ 34757us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.599212 (+ 2287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049593}
W20251214 20:52:12.599715 520 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a10663bd306d40e7a6375d06d69be22d consensus state has no leader
I20251214 20:52:12.599820 1208 tablet_replica.cc:333] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:12.600049 1208 raft_consensus.cc:2243] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:12.600135 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 7bfb575249b8403991d21c6fc96eefeb (table default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3]) successfully deleted
I20251214 20:52:12.600314 1208 pending_rounds.cc:70] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:12.600430 1208 pending_rounds.cc:77] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493699401752576 op_type: NO_OP noop_request { }
I20251214 20:52:12.600579 1208 raft_consensus.cc:2889] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:12.600713 1208 raft_consensus.cc:2272] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.600841 1205 raft_consensus.cc:2243] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.601027 1205 raft_consensus.cc:2272] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.601738 1207 raft_consensus.cc:2243] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.604031 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_5057f83c247b49baac131eb29130d8f3" } new_table_name: "default.Table_616b5b16d08b43eb8a04a21a352031fe" modify_external_catalogs: true
I20251214 20:52:12.605679 1203 raft_consensus.cc:359] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.605954 1203 raft_consensus.cc:385] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.606149 1203 raft_consensus.cc:740] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.607463 1203 consensus_queue.cc:260] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.611722 1202 tablet_bootstrap.cc:492] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:12.611915 1203 ts_tablet_manager.cc:1434] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251214 20:52:12.611970 1202 ts_tablet_manager.cc:1403] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.122s user 0.006s sys 0.004s
I20251214 20:52:12.612722 1208 ts_tablet_manager.cc:1916] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.613034 1202 raft_consensus.cc:359] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.613327 1202 raft_consensus.cc:385] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.613473 1202 raft_consensus.cc:740] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.613853 1202 consensus_queue.cc:260] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.620055 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.612313 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.622303 1203 tablet_bootstrap.cc:492] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:12.624533 1203 tablet_bootstrap.cc:654] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.631222 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.633134 1202 ts_tablet_manager.cc:1434] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251214 20:52:12.634819 960 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:12.636207 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_db08ad4da8954cba8aebd4c00ee50cd2" } modify_external_catalogs: true
I20251214 20:52:12.636463 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_db08ad4da8954cba8aebd4c00ee50cd2" } modify_external_catalogs: true
I20251214 20:52:12.638187 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.639191 1207 raft_consensus.cc:2272] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.639883 1207 ts_tablet_manager.cc:1916] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.643419 1203 tablet_bootstrap.cc:492] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:12.643729 1203 ts_tablet_manager.cc:1403] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20251214 20:52:12.644918 1415 raft_consensus.cc:493] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.645071 1415 raft_consensus.cc:515] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.645735 1415 leader_election.cc:290] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.646265 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebedf28e17a741169e5490822a2d95df" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:12.646675 915 raft_consensus.cc:2468] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:12.647235 712 leader_election.cc:304] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:12.648398 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebedf28e17a741169e5490822a2d95df" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.650290 1203 raft_consensus.cc:359] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.651242 1203 raft_consensus.cc:385] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.651403 1203 raft_consensus.cc:740] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.651723 1203 consensus_queue.cc:260] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.653079 1208 ts_tablet_manager.cc:1929] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.654459 1203 ts_tablet_manager.cc:1434] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:12.654739 1208 log.cc:1199] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/5f7aff79ce154385a73c37b10b9c2430
I20251214 20:52:12.655184 1202 tablet_bootstrap.cc:492] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:12.657163 1202 tablet_bootstrap.cc:654] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:12.659497 713 leader_election.cc:343] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.659970 1415 raft_consensus.cc:2804] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:12.660096 1415 raft_consensus.cc:493] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:12.660179 1415 raft_consensus.cc:3060] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:12.662616 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d74d3abc45a41969289c0a0892bfa04 consensus state has no leader
I20251214 20:52:12.662765 1415 raft_consensus.cc:515] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.654459 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.664947 1203 tablet_bootstrap.cc:492] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:12.666249 1415 leader_election.cc:290] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.666415 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebedf28e17a741169e5490822a2d95df" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:12.667192 1208 ts_tablet_manager.cc:1950] T 5f7aff79ce154385a73c37b10b9c2430 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
W20251214 20:52:12.667593 713 leader_election.cc:343] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.668237 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1437581b4e464656a47dc3ab99914c22" } modify_external_catalogs: true
I20251214 20:52:12.668249 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebedf28e17a741169e5490822a2d95df" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:12.668443 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1437581b4e464656a47dc3ab99914c22" } modify_external_catalogs: true
I20251214 20:52:12.668473 915 raft_consensus.cc:3060] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.669392 889 tablet_service.cc:1552] Processing DeleteTablet for tablet ebedf28e17a741169e5490822a2d95df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.673571 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 126) took 1222 ms. Trace:
I20251214 20:52:12.674180 915 raft_consensus.cc:2468] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:12.675354 523 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 118dd44151a34ba3ae58f9fe1d3790d4
I20251214 20:52:12.676185 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 118dd44151a34ba3ae58f9fe1d3790d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.676337 1208 rpcz_store.cc:276] 1214 20:52:11.451564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:11.451789 (+ 225us) service_pool.cc:225] Handling call
1214 20:52:12.599747 (+1147958us) ts_tablet_manager.cc:1151] Deleting tablet 5f7aff79ce154385a73c37b10b9c2430
1214 20:52:12.599760 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.631695 (+ 31935us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.652889 (+ 21194us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.673526 (+ 20637us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147712}
I20251214 20:52:12.676637 1342 raft_consensus.cc:493] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.676738 1208 tablet_replica.cc:333] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:12.676669 712 leader_election.cc:304] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:12.676829 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 5f7aff79ce154385a73c37b10b9c2430 (table default.Table_170ec365d121403eaaa13bb81ada4535 [id=4b90d1a1fbf649ce9406db32b838ed5b]) successfully deleted
I20251214 20:52:12.676807 1342 raft_consensus.cc:515] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.676965 1208 raft_consensus.cc:2243] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:12.677131 1415 raft_consensus.cc:2804] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:12.677234 1208 pending_rounds.cc:70] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:12.677289 1415 raft_consensus.cc:697] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:12.677322 1208 pending_rounds.cc:77] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493698485944320 op_type: NO_OP noop_request { }
I20251214 20:52:12.677454 1208 raft_consensus.cc:2889] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:12.677522 1342 leader_election.cc:290] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.677589 1208 raft_consensus.cc:2272] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.677580 1415 consensus_queue.cc:237] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.679041 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.679334 1208 ts_tablet_manager.cc:1916] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.681581 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8140dad1fab142d98b930b53ae5a64c2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:12.681787 777 raft_consensus.cc:2468] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:12.682292 848 leader_election.cc:304] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:12.682654 1342 raft_consensus.cc:2804] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:12.682777 1342 raft_consensus.cc:493] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:12.682777 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 118dd44151a34ba3ae58f9fe1d3790d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.682890 1342 raft_consensus.cc:3060] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.684978 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_4ad0cc66006940d2abacd388e227075a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.685603 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ad0cc66006940d2abacd388e227075a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.685940 1342 raft_consensus.cc:515] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.686286 1208 ts_tablet_manager.cc:1929] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.686463 1208 log.cc:1199] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/0c344296be0549e88d4811103eb6fee5
I20251214 20:52:12.686950 1208 ts_tablet_manager.cc:1950] T 0c344296be0549e88d4811103eb6fee5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:12.687391 1342 leader_election.cc:290] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.687902 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 127) took 1220 ms. Trace:
I20251214 20:52:12.688130 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8140dad1fab142d98b930b53ae5a64c2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:12.688344 777 raft_consensus.cc:3060] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.688405 1208 rpcz_store.cc:276] 1214 20:52:11.467457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:11.467770 (+ 313us) service_pool.cc:225] Handling call
1214 20:52:12.676654 (+1208884us) ts_tablet_manager.cc:1151] Deleting tablet 0c344296be0549e88d4811103eb6fee5
1214 20:52:12.676666 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.682490 (+ 5824us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.686092 (+ 3602us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.687861 (+ 1769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1208640}
I20251214 20:52:12.688879 1208 tablet_replica.cc:333] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:12.689134 1208 raft_consensus.cc:2243] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:12.689441 1208 pending_rounds.cc:70] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:12.689548 1208 pending_rounds.cc:77] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493700688977920 op_type: NO_OP noop_request { }
I20251214 20:52:12.689700 1208 raft_consensus.cc:2889] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:12.689849 1208 raft_consensus.cc:2272] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.690791 1208 ts_tablet_manager.cc:1916] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.691105 777 raft_consensus.cc:2468] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:12.691972 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 0c344296be0549e88d4811103eb6fee5 (table default.Table_3c3f2d9e5c2b47ed8038a7402612b093 [id=163e1e9d48bc4f51b308383354454363]) successfully deleted
I20251214 20:52:12.692145 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet c25832da5c1644329e18653000f3330e
I20251214 20:52:12.692503 848 leader_election.cc:304] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:12.692603 519 catalog_manager.cc:6724] ReplaceTablet: tablet c25832da5c1644329e18653000f3330e deleted and replaced by tablet 0c26d1f969ed4071a2f2cf1615a17cac
I20251214 20:52:12.694203 889 tablet_service.cc:1552] Processing DeleteTablet for tablet c25832da5c1644329e18653000f3330e with delete_type TABLET_DATA_DELETED (replaced by tablet 0c26d1f969ed4071a2f2cf1615a17cac) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.694559 755 tablet_service.cc:1552] Processing DeleteTablet for tablet c25832da5c1644329e18653000f3330e with delete_type TABLET_DATA_DELETED (replaced by tablet 0c26d1f969ed4071a2f2cf1615a17cac) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.694604 1342 raft_consensus.cc:2804] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:12.694881 1342 raft_consensus.cc:697] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:12.695253 1342 consensus_queue.cc:237] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.696765 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.698370 1208 ts_tablet_manager.cc:1929] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.698555 1208 log.cc:1199] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/a05f1ec28fb24508ae937c70aaaefffe
I20251214 20:52:12.699019 1208 ts_tablet_manager.cc:1950] T a05f1ec28fb24508ae937c70aaaefffe P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:12.699455 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 134) took 1037 ms. Trace:
I20251214 20:52:12.699527 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2162a5b765254842bb0b7e4ba7f66136" } modify_external_catalogs: true
I20251214 20:52:12.699752 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2162a5b765254842bb0b7e4ba7f66136" } modify_external_catalogs: true
I20251214 20:52:12.699612 1208 rpcz_store.cc:276] 1214 20:52:11.662510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:11.665267 (+ 2757us) service_pool.cc:225] Handling call
1214 20:52:12.688808 (+1023541us) ts_tablet_manager.cc:1151] Deleting tablet a05f1ec28fb24508ae937c70aaaefffe
1214 20:52:12.688819 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.694481 (+ 5662us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.698095 (+ 3614us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.699416 (+ 1321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1023058}
I20251214 20:52:12.700296 1208 tablet_replica.cc:333] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:12.700529 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet a05f1ec28fb24508ae937c70aaaefffe (table default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d]) successfully deleted
I20251214 20:52:12.700601 1208 raft_consensus.cc:2243] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.700804 1208 raft_consensus.cc:2272] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.701998 1208 ts_tablet_manager.cc:1916] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.702742 1206 tablet_bootstrap.cc:492] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:12.677268 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 118dd44151a34ba3ae58f9fe1d3790d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.677970 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8140dad1fab142d98b930b53ae5a64c2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.704623 1056 raft_consensus.cc:2468] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:12.683198 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.687199 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8140dad1fab142d98b930b53ae5a64c2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:12.708316 1055 raft_consensus.cc:3060] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.708803 1208 ts_tablet_manager.cc:1929] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.709743 1208 log.cc:1199] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/f44e7223f79041ec9d66d0a1bdcd4ff5
I20251214 20:52:12.710084 1208 ts_tablet_manager.cc:1950] T f44e7223f79041ec9d66d0a1bdcd4ff5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:12.710444 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 133) took 1060 ms. Trace:
I20251214 20:52:12.710528 1208 rpcz_store.cc:276] 1214 20:52:11.650345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:11.670596 (+ 20251us) service_pool.cc:225] Handling call
1214 20:52:12.700216 (+1029620us) ts_tablet_manager.cc:1151] Deleting tablet f44e7223f79041ec9d66d0a1bdcd4ff5
1214 20:52:12.700227 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:12.705265 (+ 5038us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.708667 (+ 3402us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:12.710380 (+ 1713us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027759}
I20251214 20:52:12.693861 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet c25832da5c1644329e18653000f3330e with delete_type TABLET_DATA_DELETED (replaced by tablet 0c26d1f969ed4071a2f2cf1615a17cac) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.678534 1203 tablet_bootstrap.cc:654] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.713698 1055 raft_consensus.cc:2468] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:12.714226 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_031ef16eb9c246b6ab4ce3f1661376de" } new_table_name: "default.Table_ff05c589ffd8444daa99c956e92c37e6" modify_external_catalogs: true
I20251214 20:52:12.714766 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet f44e7223f79041ec9d66d0a1bdcd4ff5 (table default.Table_eb5f20f2b40e458cade7599dbd9d5aa7 [id=e5f2cef45cbf4a69977c9be3799f9330]) successfully deleted
W20251214 20:52:12.715020 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2842f1dc0c374d24946f90dac1cf8aa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2842f1dc0c374d24946f90dac1cf8aa5 already in progress: creating tablet
W20251214 20:52:12.718045 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 9523d72fd91445539d67e1eac2a80afd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9523d72fd91445539d67e1eac2a80afd already in progress: creating tablet
W20251214 20:52:12.719146 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ed1d007f1ca24b7291262708ddfca3c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed1d007f1ca24b7291262708ddfca3c3 already in progress: creating tablet
W20251214 20:52:12.720114 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet f3a2143b5508445492b970d2588cf1cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3a2143b5508445492b970d2588cf1cf already in progress: creating tablet
I20251214 20:52:12.675611 1205 ts_tablet_manager.cc:1916] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:12.724612 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c16895ca08d94b56b5c4bc9352e330ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c16895ca08d94b56b5c4bc9352e330ef already in progress: creating tablet
W20251214 20:52:12.725526 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 6bd6856b800247de896f16eee66f2827 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bd6856b800247de896f16eee66f2827 already in progress: creating tablet
W20251214 20:52:12.726213 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet b1df0dad2fb040a5a12e0401e00c87db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1df0dad2fb040a5a12e0401e00c87db already in progress: creating tablet
I20251214 20:52:12.726581 1208 tablet_replica.cc:333] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:12.726845 1208 raft_consensus.cc:2243] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.727010 1206 ts_tablet_manager.cc:1403] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.238s user 0.010s sys 0.005s
I20251214 20:52:12.727161 1208 raft_consensus.cc:2272] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.728159 1206 raft_consensus.cc:359] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.728391 1206 raft_consensus.cc:385] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.728482 1206 raft_consensus.cc:740] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.728744 1206 consensus_queue.cc:260] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.729174 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.731829 1206 ts_tablet_manager.cc:1434] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251214 20:52:12.734236 1206 tablet_bootstrap.cc:492] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:12.735400 1202 tablet_bootstrap.cc:492] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:12.735580 1202 ts_tablet_manager.cc:1403] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.100s user 0.002s sys 0.002s
I20251214 20:52:12.736241 1206 tablet_bootstrap.cc:654] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.736325 1202 raft_consensus.cc:359] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.736507 1202 raft_consensus.cc:385] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.736578 1202 raft_consensus.cc:740] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.736614 1203 tablet_bootstrap.cc:492] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:12.736866 1203 ts_tablet_manager.cc:1403] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.006s
I20251214 20:52:12.736827 1202 consensus_queue.cc:260] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.737414 1202 ts_tablet_manager.cc:1434] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:12.737464 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.737720 1202 tablet_bootstrap.cc:492] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:12.737890 1203 raft_consensus.cc:359] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.738101 1203 raft_consensus.cc:385] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.738185 1203 raft_consensus.cc:740] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.738471 1203 consensus_queue.cc:260] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.739068 1203 ts_tablet_manager.cc:1434] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:12.739428 1202 tablet_bootstrap.cc:654] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.740196 1203 tablet_bootstrap.cc:492] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:12.740549 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.742681 1203 tablet_bootstrap.cc:654] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.743165 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 8140dad1fab142d98b930b53ae5a64c2
I20251214 20:52:12.746394 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_9e89276f2f2b4a7981f91242bdb24a87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:12.748157 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 8140dad1fab142d98b930b53ae5a64c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.748914 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 8140dad1fab142d98b930b53ae5a64c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.753247 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_5275181d46dc4a85b27fc0c45fa587fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.754068 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5275181d46dc4a85b27fc0c45fa587fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.754139 889 tablet_service.cc:1552] Processing DeleteTablet for tablet ecb69e385bf34709b5d954de4b660087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.755499 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.755617 1203 tablet_bootstrap.cc:492] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:12.755823 1203 ts_tablet_manager.cc:1403] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.002s
I20251214 20:52:12.756796 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 8140dad1fab142d98b930b53ae5a64c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.756799 751 tablet_service.cc:1552] Processing DeleteTablet for tablet ecb69e385bf34709b5d954de4b660087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.757397 1203 raft_consensus.cc:359] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:12.757695 1203 raft_consensus.cc:385] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.758678 1203 raft_consensus.cc:740] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.759472 1203 consensus_queue.cc:260] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:12.760094 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.760344 1203 ts_tablet_manager.cc:1434] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251214 20:52:12.760696 1203 tablet_bootstrap.cc:492] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:12.767118 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e89276f2f2b4a7981f91242bdb24a87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.770008 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.774170 525 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 33a7f355b40f472883e3399209ef6762 (table default.Table_1c75ca10deb842669e2fea0f16ecc983 [id=5272550343144d4baaba07e97ebaa187]). Expected version 1 got 0
I20251214 20:52:12.775404 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.775583 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1fe00d64c7144563a94dac2fb9a0b29f" } new_table_name: "default.Table_67ec8d0cd3f948218b4637de6d68a57b" modify_external_catalogs: true
W20251214 20:52:12.776896 520 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b48bb260e854beb9ec61ce9c79b46c2 consensus state has no leader
I20251214 20:52:12.777277 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_33ca298479304caaa7c59e715fd082b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.777774 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33ca298479304caaa7c59e715fd082b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.778873 517 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet f31583af84ff434eba07f361775a753c
I20251214 20:52:12.779264 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_727db60cbb7c435d986f772214b16166"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.779772 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_727db60cbb7c435d986f772214b16166 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.780959 526 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 12b5f4326aa0413eb303c132ba21decd
I20251214 20:52:12.781672 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.782231 523 master_service.cc:954] ReplaceTablet: received request to replace tablet b1df0dad2fb040a5a12e0401e00c87db from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.782737 523 master_service.cc:954] ReplaceTablet: received request to replace tablet a8e09c13caca4a1c9018387972c9f082 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.783259 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_53273de8ee634ce9994c46fe98bba85e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.783830 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53273de8ee634ce9994c46fe98bba85e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.785043 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 84a0237e4ee34f94821d8647a400575b
I20251214 20:52:12.789863 1342 raft_consensus.cc:493] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.790076 1342 raft_consensus.cc:515] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.790830 1342 leader_election.cc:290] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.791746 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c17126f75b6241c5833269d20774c439" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:12.792419 848 leader_election.cc:343] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.793309 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c17126f75b6241c5833269d20774c439" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.793751 519 master_service.cc:954] ReplaceTablet: received request to replace tablet b652a8a80246443491551e76e68d2a8b from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:12.794174 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 12b5f4326aa0413eb303c132ba21decd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.794409 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 84a0237e4ee34f94821d8647a400575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.795812 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 12b5f4326aa0413eb303c132ba21decd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.796144 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 84a0237e4ee34f94821d8647a400575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.797731 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 84a0237e4ee34f94821d8647a400575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.799491 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 12b5f4326aa0413eb303c132ba21decd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.799801 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ba2d34a6d2645bd97fe85fed952692b with delete_type TABLET_DATA_DELETED (replaced by tablet f0d661a1bb034053a4634ff537eafc7d) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.800024 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet ecb69e385bf34709b5d954de4b660087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:12.800366 851 leader_election.cc:343] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.800549 851 leader_election.cc:304] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:12.801106 1342 raft_consensus.cc:2749] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:12.801621 1203 tablet_bootstrap.cc:654] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:12.802273 522 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 324379e3217442c3addcfe58060b9ad7 has no consensus state
I20251214 20:52:12.802666 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a30f394f4b6e4b3385cb6abafe21b8aa" } new_table_name: "default.Table_d9cef6f07b8743c3aa470da0be52c7c0" modify_external_catalogs: true
I20251214 20:52:12.804586 1206 tablet_bootstrap.cc:492] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:12.805770 1208 ts_tablet_manager.cc:1916] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.808531 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet d8947ebdc87341ad8b1a01f5dd246172 with delete_type TABLET_DATA_DELETED (replaced by tablet d13f93bb5e5f4b59bcf9f4bfed070f25) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.805209 1206 ts_tablet_manager.cc:1403] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.013s
I20251214 20:52:12.809950 1206 raft_consensus.cc:359] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.810222 1206 raft_consensus.cc:385] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.810366 1206 raft_consensus.cc:740] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.810729 1206 consensus_queue.cc:260] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.811323 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.812427 1342 raft_consensus.cc:493] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.812675 1342 raft_consensus.cc:515] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.813274 1342 leader_election.cc:290] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.815608 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8e09c13caca4a1c9018387972c9f082" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.816532 1202 tablet_bootstrap.cc:492] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:12.816504 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8e09c13caca4a1c9018387972c9f082" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:12.816725 1202 ts_tablet_manager.cc:1403] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.005s
W20251214 20:52:12.817247 848 leader_election.cc:343] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:12.817672 851 leader_election.cc:343] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.817636 1202 raft_consensus.cc:359] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.817838 1202 raft_consensus.cc:385] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.817814 851 leader_election.cc:304] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:12.817926 1202 raft_consensus.cc:740] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.818141 1202 consensus_queue.cc:260] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.818504 1213 raft_consensus.cc:2749] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:12.818686 1202 ts_tablet_manager.cc:1434] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:12.818989 1202 tablet_bootstrap.cc:492] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:12.819641 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.821498 1202 tablet_bootstrap.cc:654] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.825311 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3047cd5c72e34614924729dcc11eb7f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.825953 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3047cd5c72e34614924729dcc11eb7f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.831338 882 tablet_service.cc:1552] Processing DeleteTablet for tablet f2b66900d6ea42aa849c24acbbe23c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:12.836469 1206 ts_tablet_manager.cc:1434] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20251214 20:52:12.836329 1448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:12.836750 1448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:12.836335 1434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6713a20a82b434f8a8bc122113e26ad. Latest consensus state: current_term: 3 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:12.836993 1434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:12.837389 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
W20251214 20:52:12.845101 518 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9b7149a436a4cab80b39adccf8567f9 has no consensus state
I20251214 20:52:12.846886 755 tablet_service.cc:1552] Processing DeleteTablet for tablet c17126f75b6241c5833269d20774c439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:12.852200 1202 tablet_bootstrap.cc:492] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:12.852559 1202 ts_tablet_manager.cc:1403] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.005s
I20251214 20:52:12.853363 1206 tablet_bootstrap.cc:492] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:12.854908 1202 raft_consensus.cc:359] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:12.855289 1202 raft_consensus.cc:385] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.855406 1202 raft_consensus.cc:740] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.855758 1202 consensus_queue.cc:260] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:12.856372 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.858987 1202 ts_tablet_manager.cc:1434] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:12.859301 1415 raft_consensus.cc:493] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.859515 1202 tablet_bootstrap.cc:492] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:12.859656 1415 raft_consensus.cc:515] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.860384 1415 leader_election.cc:290] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.861249 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ecb69e385bf34709b5d954de4b660087" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:12.861541 1055 raft_consensus.cc:2468] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:12.862007 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ecb69e385bf34709b5d954de4b660087" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:12.862275 915 raft_consensus.cc:2468] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:12.863600 712 leader_election.cc:304] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:12.864841 1415 raft_consensus.cc:2804] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:12.864956 1415 raft_consensus.cc:493] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:12.865053 1415 raft_consensus.cc:3060] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.865447 1176 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:12.865722 1176 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:12.867805 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_3900abc2f8dc49ceb70dc23666566d3f" } new_table_name: "default.Table_934b554f3bb44b5e91119f9b78acb150" modify_external_catalogs: true
I20251214 20:52:12.868783 1415 raft_consensus.cc:515] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.868932 1207 ts_tablet_manager.cc:1929] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.869102 1207 log.cc:1199] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/5ba2d34a6d2645bd97fe85fed952692b
I20251214 20:52:12.869503 1207 ts_tablet_manager.cc:1950] T 5ba2d34a6d2645bd97fe85fed952692b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:12.870186 1345 raft_consensus.cc:493] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.870317 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ecb69e385bf34709b5d954de4b660087" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:12.870399 1345 raft_consensus.cc:515] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:12.870558 915 raft_consensus.cc:3060] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.870785 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 5ba2d34a6d2645bd97fe85fed952692b (table default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90]) successfully deleted
I20251214 20:52:12.871203 1345 leader_election.cc:290] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:12.871810 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8947ebdc87341ad8b1a01f5dd246172" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:12.871904 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d8947ebdc87341ad8b1a01f5dd246172" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:12.872092 777 raft_consensus.cc:2393] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63848a9fb78a4013a6f1657621272d75 in current term 1: Already voted for candidate a2e8fd1cb731446d88f8e510c7a52999 in this term.
I20251214 20:52:12.872570 991 leader_election.cc:304] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:12.872696 1206 tablet_bootstrap.cc:654] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.873003 1345 raft_consensus.cc:3060] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.873430 915 raft_consensus.cc:2468] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:12.873970 712 leader_election.cc:304] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:12.874300 1341 raft_consensus.cc:2804] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:12.876055 1345 raft_consensus.cc:2749] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:12.881858 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.882680 1415 leader_election.cc:290] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.883255 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ecb69e385bf34709b5d954de4b660087" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:12.883530 1055 raft_consensus.cc:3060] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.883596 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.886076 1205 ts_tablet_manager.cc:1929] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:12.887781 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a519bf6d1ab8419784be3772e899869d" } modify_external_catalogs: true
I20251214 20:52:12.887984 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a519bf6d1ab8419784be3772e899869d" } modify_external_catalogs: true
I20251214 20:52:12.899834 1341 raft_consensus.cc:697] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:12.900276 1341 consensus_queue.cc:237] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.904680 1208 ts_tablet_manager.cc:1929] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:12.904951 1208 log.cc:1199] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/44f51e11041841e98b9a15d380226e4e
I20251214 20:52:12.905509 1207 tablet_replica.cc:333] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:12.905807 1207 raft_consensus.cc:2243] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:12.906078 1207 raft_consensus.cc:2272] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:12.906751 1207 ts_tablet_manager.cc:1916] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:12.906982 1202 tablet_bootstrap.cc:654] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.908196 1460 raft_consensus.cc:493] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.908433 1460 raft_consensus.cc:515] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:12.909430 522 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 144dee1a6d7344f2b79da6e79e67599e consensus state has no leader
I20251214 20:52:12.909790 1460 leader_election.cc:290] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.913751 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_181aab73cbf1461b85364fc893700db7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:12.914322 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_181aab73cbf1461b85364fc893700db7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:12.915752 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2b66900d6ea42aa849c24acbbe23c40" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:12.915998 915 raft_consensus.cc:2468] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:12.916460 712 leader_election.cc:304] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:12.916815 1460 raft_consensus.cc:2804] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:12.916927 1460 raft_consensus.cc:493] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:12.917019 1460 raft_consensus.cc:3060] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.919533 1206 tablet_bootstrap.cc:492] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:12.919731 1206 ts_tablet_manager.cc:1403] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.000s
I20251214 20:52:12.920481 1460 raft_consensus.cc:515] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.922099 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2b66900d6ea42aa849c24acbbe23c40" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:12.922354 915 raft_consensus.cc:3060] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.886269 1055 raft_consensus.cc:2468] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:12.889309 1203 tablet_bootstrap.cc:492] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:12.923583 1203 ts_tablet_manager.cc:1403] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.163s user 0.002s sys 0.002s
I20251214 20:52:12.910641 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2b66900d6ea42aa849c24acbbe23c40" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:12.924539 713 leader_election.cc:343] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.924906 1203 raft_consensus.cc:359] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.925061 1203 raft_consensus.cc:385] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.925145 1203 raft_consensus.cc:740] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.925354 1203 consensus_queue.cc:260] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.925880 1203 ts_tablet_manager.cc:1434] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:12.926048 1415 consensus_queue.cc:1048] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:12.926179 1203 tablet_bootstrap.cc:492] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:12.927553 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.927837 915 raft_consensus.cc:2468] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:12.927882 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.928532 712 leader_election.cc:304] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:12.928896 1415 raft_consensus.cc:2804] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:12.932195 524 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 3fd0fa0dad974c98bf6f52c8cfe83035 (table default.Table_0173f37106d54db18ef3975bc6b0d185 [id=5c14810dd5364dcaa218b80c3756eadc]). Expected version 1 got 0
I20251214 20:52:12.934072 1022 tablet_service.cc:1505] Processing CreateTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf (DEFAULT_TABLE table=default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.934533 1203 tablet_bootstrap.cc:654] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.934626 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34eb8fa3e34427495fa5985b7ba81bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.935582 755 tablet_service.cc:1505] Processing CreateTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf (DEFAULT_TABLE table=default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.935650 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2b66900d6ea42aa849c24acbbe23c40" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:12.936033 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34eb8fa3e34427495fa5985b7ba81bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:12.939548 713 leader_election.cc:343] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:12.944428 882 tablet_service.cc:1505] Processing CreateTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf (DEFAULT_TABLE table=default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.943634 1460 leader_election.cc:290] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:12.944995 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34eb8fa3e34427495fa5985b7ba81bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.945425 1203 tablet_bootstrap.cc:492] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:12.945596 1203 ts_tablet_manager.cc:1403] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20251214 20:52:12.946415 1203 raft_consensus.cc:359] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.946606 1203 raft_consensus.cc:385] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.946674 1203 raft_consensus.cc:740] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.946914 1203 consensus_queue.cc:260] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.950148 1203 ts_tablet_manager.cc:1434] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:12.951045 1345 raft_consensus.cc:493] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:12.951303 1345 raft_consensus.cc:515] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.952090 1345 leader_election.cc:290] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:12.952864 1101 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:12.952872 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ba2d34a6d2645bd97fe85fed952692b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:12.953022 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:12.953182 915 raft_consensus.cc:2468] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:12.953670 991 leader_election.cc:304] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:12.954123 1345 raft_consensus.cc:2804] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:12.954212 1345 raft_consensus.cc:493] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:12.954303 1345 raft_consensus.cc:3060] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:12.954483 989 leader_election.cc:343] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 5ba2d34a6d2645bd97fe85fed952692b
I20251214 20:52:12.955315 1203 tablet_bootstrap.cc:492] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:12.957269 1203 tablet_bootstrap.cc:654] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.957967 751 tablet_service.cc:1505] Processing CreateTablet for tablet bf37c50e511f454ca4887c7f6ca1a746 (DEFAULT_TABLE table=default.Table_0071d629c0444fa89511b0938f2e757e [id=4616162e7f9c4c55a4dd98c375d2c2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.958166 1206 raft_consensus.cc:359] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.958461 1206 raft_consensus.cc:385] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:12.958956 889 tablet_service.cc:1505] Processing CreateTablet for tablet bf37c50e511f454ca4887c7f6ca1a746 (DEFAULT_TABLE table=default.Table_0071d629c0444fa89511b0938f2e757e [id=4616162e7f9c4c55a4dd98c375d2c2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.959419 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf37c50e511f454ca4887c7f6ca1a746. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.961621 749 tablet_service.cc:1505] Processing CreateTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 (DEFAULT_TABLE table=default.Table_d863e40d2a9e46649f9dc97f33a616db [id=cd65155def8244e59f08dac4d695567d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.962634 882 tablet_service.cc:1505] Processing CreateTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 (DEFAULT_TABLE table=default.Table_d863e40d2a9e46649f9dc97f33a616db [id=cd65155def8244e59f08dac4d695567d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.958564 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf37c50e511f454ca4887c7f6ca1a746. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.963147 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c0f445cd7e44769d79ad5566fd8d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.965539 755 tablet_service.cc:1505] Processing CreateTablet for tablet 24ff58f321124d12b5330dca185d426e (DEFAULT_TABLE table=default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.965950 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ff58f321124d12b5330dca185d426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.971033 742 tablet_service.cc:1505] Processing CreateTablet for tablet 324379e3217442c3addcfe58060b9ad7 (DEFAULT_TABLE table=default.Table_ff05c589ffd8444daa99c956e92c37e6 [id=16a8869e1ba641438d0836f6f204414f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.971468 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 324379e3217442c3addcfe58060b9ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.971464 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ade67d7e10e04c6fb0cfc134947a559d" } modify_external_catalogs: true
I20251214 20:52:12.971707 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ade67d7e10e04c6fb0cfc134947a559d" } modify_external_catalogs: true
I20251214 20:52:12.975556 888 tablet_service.cc:1505] Processing CreateTablet for tablet 24ff58f321124d12b5330dca185d426e (DEFAULT_TABLE table=default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.958557 1206 raft_consensus.cc:740] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:12.981189 524 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_616b5b16d08b43eb8a04a21a352031fe [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 1 got 0
I20251214 20:52:12.981232 1206 consensus_queue.cc:260] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.981979 1206 ts_tablet_manager.cc:1434] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.062s user 0.000s sys 0.003s
I20251214 20:52:12.982574 880 tablet_service.cc:1505] Processing CreateTablet for tablet 324379e3217442c3addcfe58060b9ad7 (DEFAULT_TABLE table=default.Table_ff05c589ffd8444daa99c956e92c37e6 [id=16a8869e1ba641438d0836f6f204414f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.982759 1206 tablet_bootstrap.cc:492] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:12.983251 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c0f445cd7e44769d79ad5566fd8d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.983690 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 324379e3217442c3addcfe58060b9ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.984040 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ff58f321124d12b5330dca185d426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.985014 1206 tablet_bootstrap.cc:654] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:12.986248 1208 ts_tablet_manager.cc:1950] T 44f51e11041841e98b9a15d380226e4e P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:12.986539 742 tablet_service.cc:1505] Processing CreateTablet for tablet c9b7149a436a4cab80b39adccf8567f9 (DEFAULT_TABLE table=default.Table_67ec8d0cd3f948218b4637de6d68a57b [id=bdc2d86a977d4146a1a86717c5916807]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.986915 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b7149a436a4cab80b39adccf8567f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.986891 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 44f51e11041841e98b9a15d380226e4e (table default.Table_0191351f0cd945e489dc8d02dbe2c5be [id=3aaa67725e8442fdbae975274c8596c4]) successfully deleted
I20251214 20:52:12.987291 1208 tablet_replica.cc:333] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:12.957357 1345 raft_consensus.cc:515] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:12.991936 1345 leader_election.cc:290] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:12.992743 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ba2d34a6d2645bd97fe85fed952692b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:12.992975 915 raft_consensus.cc:3060] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:12.962749 1035 tablet_service.cc:1505] Processing CreateTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 (DEFAULT_TABLE table=default.Table_d863e40d2a9e46649f9dc97f33a616db [id=cd65155def8244e59f08dac4d695567d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.995586 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6c0f445cd7e44769d79ad5566fd8d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.997447 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_bff6195ce6594ee9b4a55eb6058dcb63" } new_table_name: "default.Table_5fbbdeeea9a14e17bf6f9d0cc4a56d98" modify_external_catalogs: true
I20251214 20:52:12.997882 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 07a8662a634a4befbd2d6251000d2059 (DEFAULT_TABLE table=default.Table_1a95326a5ce4436ca97f62d32966ec22 [id=6157817e96d642d09f9941ad7f7e82ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.998541 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a8662a634a4befbd2d6251000d2059. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.967423 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 24ff58f321124d12b5330dca185d426e (DEFAULT_TABLE table=default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.998950 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ff58f321124d12b5330dca185d426e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.999372 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_c5d8c1bdf3344ec5a0d409a1abf98b45" } modify_external_catalogs: true
I20251214 20:52:12.999598 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_c5d8c1bdf3344ec5a0d409a1abf98b45" } modify_external_catalogs: true
W20251214 20:52:13.002789 525 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd8226b431c6456885e928880938969d consensus state has no leader
I20251214 20:52:12.969033 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 324379e3217442c3addcfe58060b9ad7 (DEFAULT_TABLE table=default.Table_ff05c589ffd8444daa99c956e92c37e6 [id=16a8869e1ba641438d0836f6f204414f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:12.974038 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet ebedf28e17a741169e5490822a2d95df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:12.980072 1029 tablet_service.cc:1505] Processing CreateTablet for tablet c9b7149a436a4cab80b39adccf8567f9 (DEFAULT_TABLE table=default.Table_67ec8d0cd3f948218b4637de6d68a57b [id=bdc2d86a977d4146a1a86717c5916807]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.003553 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 324379e3217442c3addcfe58060b9ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.003808 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b7149a436a4cab80b39adccf8567f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.007622 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2325a2c5e97f45ebac4c943380bb3525" } modify_external_catalogs: true
I20251214 20:52:13.007867 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2325a2c5e97f45ebac4c943380bb3525" } modify_external_catalogs: true
I20251214 20:52:13.009186 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_caa5b0dabd9d4ea8a8b16c550407dc53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.009799 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_caa5b0dabd9d4ea8a8b16c550407dc53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.012557 915 raft_consensus.cc:2468] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:13.012871 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_c9b7a105aebf4ed48e4162d811977686"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:13.013084 991 leader_election.cc:304] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:13.013463 1345 raft_consensus.cc:2804] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
W20251214 20:52:13.013645 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9b7a105aebf4ed48e4162d811977686 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.017014 1345 raft_consensus.cc:697] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:13.017328 1345 consensus_queue.cc:237] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.017686 880 tablet_service.cc:1505] Processing CreateTablet for tablet 07a8662a634a4befbd2d6251000d2059 (DEFAULT_TABLE table=default.Table_1a95326a5ce4436ca97f62d32966ec22 [id=6157817e96d642d09f9941ad7f7e82ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.018222 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a8662a634a4befbd2d6251000d2059. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:12.980742 1022 tablet_service.cc:1505] Processing CreateTablet for tablet bf37c50e511f454ca4887c7f6ca1a746 (DEFAULT_TABLE table=default.Table_0071d629c0444fa89511b0938f2e757e [id=4616162e7f9c4c55a4dd98c375d2c2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.019753 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf37c50e511f454ca4887c7f6ca1a746. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.024065 889 tablet_service.cc:1505] Processing CreateTablet for tablet c9b7149a436a4cab80b39adccf8567f9 (DEFAULT_TABLE table=default.Table_67ec8d0cd3f948218b4637de6d68a57b [id=bdc2d86a977d4146a1a86717c5916807]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.024801 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b7149a436a4cab80b39adccf8567f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.028872 1202 tablet_bootstrap.cc:492] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:12.886231 1205 log.cc:1199] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/a05f1ec28fb24508ae937c70aaaefffe
I20251214 20:52:13.029105 1202 ts_tablet_manager.cc:1403] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.170s user 0.004s sys 0.004s
I20251214 20:52:13.029760 1205 ts_tablet_manager.cc:1950] T a05f1ec28fb24508ae937c70aaaefffe P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:13.030050 1202 raft_consensus.cc:359] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.030259 1202 raft_consensus.cc:385] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.030356 1202 raft_consensus.cc:740] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.030733 1202 consensus_queue.cc:260] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.030792 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet a05f1ec28fb24508ae937c70aaaefffe (table default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d]) successfully deleted
I20251214 20:52:13.031280 1202 ts_tablet_manager.cc:1434] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:13.031587 1202 tablet_bootstrap.cc:492] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:13.031778 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 60ae3db968f6412aa0c3952fafe65f88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60ae3db968f6412aa0c3952fafe65f88 already in progress: creating tablet
W20251214 20:52:13.032470 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 376e5efede87410eb9f95f36bedebf9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 376e5efede87410eb9f95f36bedebf9d already in progress: creating tablet
W20251214 20:52:13.033175 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet d768c45bc02b4d108798efbf3f6aaf7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d768c45bc02b4d108798efbf3f6aaf7d already in progress: creating tablet
I20251214 20:52:13.036566 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a5388ce81c77421aa5c30bfe834f60d3" } modify_external_catalogs: true
I20251214 20:52:13.037024 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a5388ce81c77421aa5c30bfe834f60d3" } modify_external_catalogs: true
I20251214 20:52:13.037904 1205 tablet_replica.cc:333] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:13.038254 1205 raft_consensus.cc:2243] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.038595 1205 raft_consensus.cc:2272] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.043332 1208 raft_consensus.cc:2243] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.044368 1208 pending_rounds.cc:70] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:13.044597 1208 pending_rounds.cc:77] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493699388424192 op_type: NO_OP noop_request { }
I20251214 20:52:13.044817 1208 raft_consensus.cc:2889] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:13.046635 1208 raft_consensus.cc:2272] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.046046 888 tablet_service.cc:1505] Processing CreateTablet for tablet 604b7c6fe3b24daca70711a1990af556 (DEFAULT_TABLE table=default.Table_fbb5b21626c142c8baac60a7f29d3c51 [id=7cb78d704474480db14d212ee89166a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.048648 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604b7c6fe3b24daca70711a1990af556. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.049793 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.050626 526 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b
I20251214 20:52:13.051106 526 catalog_manager.cc:6724] ReplaceTablet: tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b deleted and replaced by tablet e8c02a2e4a2640b1b3a69f25a708684b
I20251214 20:52:13.053297 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b with delete_type TABLET_DATA_DELETED (replaced by tablet e8c02a2e4a2640b1b3a69f25a708684b) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.054523 1202 tablet_bootstrap.cc:654] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.059213 526 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 3fd0fa0dad974c98bf6f52c8cfe83035 (table default.Table_0173f37106d54db18ef3975bc6b0d185 [id=5c14810dd5364dcaa218b80c3756eadc]). Expected version 1 got 0
I20251214 20:52:13.060141 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_73d965cabee84dd3865221877063447c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.060950 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73d965cabee84dd3865221877063447c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:13.055819 989 leader_election.cc:343] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 5ba2d34a6d2645bd97fe85fed952692b
I20251214 20:52:13.062268 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b with delete_type TABLET_DATA_DELETED (replaced by tablet e8c02a2e4a2640b1b3a69f25a708684b) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.056968 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 604b7c6fe3b24daca70711a1990af556 (DEFAULT_TABLE table=default.Table_fbb5b21626c142c8baac60a7f29d3c51 [id=7cb78d704474480db14d212ee89166a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.065178 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604b7c6fe3b24daca70711a1990af556. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.065697 888 tablet_service.cc:1505] Processing CreateTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 (DEFAULT_TABLE table=default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.066287 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bbb28259c1442ae94f4eff76c25c539. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.075567 1202 tablet_bootstrap.cc:492] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:13.075978 1202 ts_tablet_manager.cc:1403] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.006s
I20251214 20:52:13.076987 1202 raft_consensus.cc:359] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.078114 1202 raft_consensus.cc:385] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.078382 1202 raft_consensus.cc:740] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.078738 1202 consensus_queue.cc:260] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.078974 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.082882 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.088476 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 (DEFAULT_TABLE table=default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.089143 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bbb28259c1442ae94f4eff76c25c539. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.092458 1202 ts_tablet_manager.cc:1434] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251214 20:52:13.094075 960 heartbeater.cc:344] Connected to a master server at 127.0.104.60:35099
I20251214 20:52:13.094281 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.096915 1205 ts_tablet_manager.cc:1916] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.096995 888 tablet_service.cc:1505] Processing CreateTablet for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (DEFAULT_TABLE table=default.Table_0191351f0cd945e489dc8d02dbe2c5be [id=3aaa67725e8442fdbae975274c8596c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.097324 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (DEFAULT_TABLE table=default.Table_0191351f0cd945e489dc8d02dbe2c5be [id=3aaa67725e8442fdbae975274c8596c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.097504 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f4760585baf4b4ab3b9b94b5a748a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.097864 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f4760585baf4b4ab3b9b94b5a748a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.098475 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet b652a8a80246443491551e76e68d2a8b
I20251214 20:52:13.098951 519 catalog_manager.cc:6724] ReplaceTablet: tablet b652a8a80246443491551e76e68d2a8b deleted and replaced by tablet 18e119bc44874aecb4e83c8d6040c8cd
I20251214 20:52:13.100085 519 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_616b5b16d08b43eb8a04a21a352031fe [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 1 got 0
I20251214 20:52:13.100752 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_520f27438b664effa96397e5bf61379d" } new_table_name: "default.Table_bda8a86e13ba427e94e8d85a318281ad" modify_external_catalogs: true
I20251214 20:52:13.101908 523 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 8af99e0bd8774a678575847a9ec2e557
I20251214 20:52:13.102309 1203 tablet_bootstrap.cc:492] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:13.102440 1203 ts_tablet_manager.cc:1403] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.147s user 0.003s sys 0.002s
I20251214 20:52:13.102591 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_ecba341e009147b9b843be246ea2cd7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.103109 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecba341e009147b9b843be246ea2cd7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.103261 1203 raft_consensus.cc:359] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.103462 1203 raft_consensus.cc:385] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.103569 1203 raft_consensus.cc:740] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.104233 1208 ts_tablet_manager.cc:1916] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.105652 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af99e0bd8774a678575847a9ec2e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.107729 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet b652a8a80246443491551e76e68d2a8b with delete_type TABLET_DATA_DELETED (replaced by tablet 18e119bc44874aecb4e83c8d6040c8cd) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.108090 1203 consensus_queue.cc:260] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.109539 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af99e0bd8774a678575847a9ec2e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.109915 882 tablet_service.cc:1552] Processing DeleteTablet for tablet b652a8a80246443491551e76e68d2a8b with delete_type TABLET_DATA_DELETED (replaced by tablet 18e119bc44874aecb4e83c8d6040c8cd) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.113093 1203 ts_tablet_manager.cc:1434] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251214 20:52:13.114437 1203 tablet_bootstrap.cc:492] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:13.116492 517 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 52daf1204bf142e7b82852bdec6582fb consensus state has no leader
I20251214 20:52:13.117019 1203 tablet_bootstrap.cc:654] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.120499 1208 ts_tablet_manager.cc:1929] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:13.120702 1208 log.cc:1199] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/358d5c6b594147c8ba136b4b0732fae1
I20251214 20:52:13.121204 1208 ts_tablet_manager.cc:1950] T 358d5c6b594147c8ba136b4b0732fae1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:13.124890 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4b9e6d9e76414e6da9e6767b3463fa10" } new_table_name: "default.Table_2342c4af4f9b420b9b7cbcf327ce1733" modify_external_catalogs: true
I20251214 20:52:13.127444 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet a8e09c13caca4a1c9018387972c9f082 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.128806 1205 ts_tablet_manager.cc:1929] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.129014 1205 log.cc:1199] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/8c3b9ff3c6234507b76afd18f097b6e0
I20251214 20:52:13.129550 1205 ts_tablet_manager.cc:1950] T 8c3b9ff3c6234507b76afd18f097b6e0 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:13.130573 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 358d5c6b594147c8ba136b4b0732fae1 (table default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]) successfully deleted
I20251214 20:52:13.131713 1342 raft_consensus.cc:493] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.131877 1342 raft_consensus.cc:515] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.132604 1342 leader_election.cc:290] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.133033 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33a7f355b40f472883e3399209ef6762" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:13.133265 1056 raft_consensus.cc:2468] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:13.134025 851 leader_election.cc:304] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:13.134279 1213 raft_consensus.cc:2804] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:13.134378 889 tablet_service.cc:1552] Processing DeleteTablet for tablet ed1d007f1ca24b7291262708ddfca3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.134387 1213 raft_consensus.cc:493] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:13.134505 1213 raft_consensus.cc:3060] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.134824 888 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3962c33a294fa98f8bd485268aee71 (DEFAULT_TABLE table=default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.135242 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3962c33a294fa98f8bd485268aee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.135990 1460 raft_consensus.cc:493] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.137243 1213 raft_consensus.cc:515] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.137847 1213 leader_election.cc:290] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.139286 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.139614 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33a7f355b40f472883e3399209ef6762" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:13.139809 1055 raft_consensus.cc:3060] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.141974 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33a7f355b40f472883e3399209ef6762" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:13.142277 776 raft_consensus.cc:2468] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:13.142315 1055 raft_consensus.cc:2468] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:13.143033 851 leader_election.cc:304] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:13.143811 1342 raft_consensus.cc:2804] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:13.144392 1213 consensus_queue.cc:1048] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:13.144826 1203 tablet_bootstrap.cc:492] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:13.145110 1203 ts_tablet_manager.cc:1403] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.003s
I20251214 20:52:13.142788 1460 raft_consensus.cc:515] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.145910 1203 raft_consensus.cc:359] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:13.146528 848 consensus_peers.cc:597] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:13.146924 1208 tablet_replica.cc:333] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:13.147217 1208 raft_consensus.cc:2243] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.147390 1208 raft_consensus.cc:2272] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:13.147455 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ba50e32f7e724868bb891045a939e362 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba50e32f7e724868bb891045a939e362 already in progress: creating tablet
W20251214 20:52:13.147774 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 60ae3db968f6412aa0c3952fafe65f88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60ae3db968f6412aa0c3952fafe65f88 already in progress: creating tablet
W20251214 20:52:13.147996 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet d768c45bc02b4d108798efbf3f6aaf7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d768c45bc02b4d108798efbf3f6aaf7d already in progress: creating tablet
W20251214 20:52:13.149226 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 376e5efede87410eb9f95f36bedebf9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 376e5efede87410eb9f95f36bedebf9d already in progress: creating tablet
I20251214 20:52:13.151253 1202 tablet_bootstrap.cc:492] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:13.153107 1202 tablet_bootstrap.cc:654] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.153555 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33a7f355b40f472883e3399209ef6762" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:13.153859 776 raft_consensus.cc:3060] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.154263 1203 raft_consensus.cc:385] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.154359 1203 raft_consensus.cc:740] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.153939 1208 ts_tablet_manager.cc:1916] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.160048 776 raft_consensus.cc:2468] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:13.161065 518 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 0c26d1f969ed4071a2f2cf1615a17cac
I20251214 20:52:13.166724 1203 consensus_queue.cc:260] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.168973 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 8d72f55d2d2440d993aa205d96c356f1 (DEFAULT_TABLE table=default.Table_2f947b9f382942e082ea4909657a8f34 [id=66d124c2a8dd4cc188a832eff1dbe9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.169076 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_04ad0003a0524c668f9118432b1b816f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:13.169427 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d72f55d2d2440d993aa205d96c356f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:13.169816 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04ad0003a0524c668f9118432b1b816f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.170603 1203 ts_tablet_manager.cc:1434] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.025s user 0.000s sys 0.003s
I20251214 20:52:13.170912 1203 tablet_bootstrap.cc:492] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:13.172621 1203 tablet_bootstrap.cc:654] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.175637 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3962c33a294fa98f8bd485268aee71 (DEFAULT_TABLE table=default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.176306 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3962c33a294fa98f8bd485268aee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.182874 880 tablet_service.cc:1505] Processing CreateTablet for tablet 8d72f55d2d2440d993aa205d96c356f1 (DEFAULT_TABLE table=default.Table_2f947b9f382942e082ea4909657a8f34 [id=66d124c2a8dd4cc188a832eff1dbe9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.194057 525 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 2da66ec83e9140f7a605f0c86abff468
I20251214 20:52:13.194872 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 2da66ec83e9140f7a605f0c86abff468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.196287 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 8c3b9ff3c6234507b76afd18f097b6e0 (table default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]) successfully deleted
W20251214 20:52:13.197237 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 118dd44151a34ba3ae58f9fe1d3790d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 118dd44151a34ba3ae58f9fe1d3790d4 already in progress: creating tablet
I20251214 20:52:13.198120 1205 tablet_replica.cc:333] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:13.198475 1205 raft_consensus.cc:2243] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.200155 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_93388fdcfc2146b1b4e0f3554a12f10f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.200834 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93388fdcfc2146b1b4e0f3554a12f10f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.158366 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.195616 1202 tablet_bootstrap.cc:492] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:13.196049 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 2da66ec83e9140f7a605f0c86abff468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.207058 1202 ts_tablet_manager.cc:1403] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.001s
I20251214 20:52:13.208441 1202 raft_consensus.cc:359] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.208700 1202 raft_consensus.cc:385] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.208838 1202 raft_consensus.cc:740] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.209415 888 tablet_service.cc:1505] Processing CreateTablet for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (DEFAULT_TABLE table=default.Table_d78784658d7d47e1b55c72607d6169b0 [id=ed335fb680bc441184e1682200ee68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.209786 1202 consensus_queue.cc:260] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.202848 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet f2b66900d6ea42aa849c24acbbe23c40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.210922 1202 ts_tablet_manager.cc:1434] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251214 20:52:13.210944 1205 raft_consensus.cc:2272] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.211476 1202 tablet_bootstrap.cc:492] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:13.198072 1208 ts_tablet_manager.cc:1929] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.211699 1205 ts_tablet_manager.cc:1916] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.211943 1208 log.cc:1199] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/5ba2d34a6d2645bd97fe85fed952692b
I20251214 20:52:13.212684 1208 ts_tablet_manager.cc:1950] T 5ba2d34a6d2645bd97fe85fed952692b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:13.213657 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 5ba2d34a6d2645bd97fe85fed952692b (table default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90]) successfully deleted
I20251214 20:52:13.214174 1208 tablet_replica.cc:333] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:13.206305 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d72f55d2d2440d993aa205d96c356f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.214856 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fcdd6bafa5c4aa6b59668cde523d010. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.219447 1342 raft_consensus.cc:697] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:13.219808 1342 consensus_queue.cc:237] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.219846 1460 leader_election.cc:290] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.220129 1460 raft_consensus.cc:493] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.220707 1460 raft_consensus.cc:515] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.221514 1460 leader_election.cc:290] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.222028 1030 tablet_service.cc:1505] Processing CreateTablet for tablet bb698ff30fcc47d09000cc68d2c5225b (DEFAULT_TABLE table=default.Table_ab4c9e0bdaf74a1791ddad541ece2482 [id=8c96acfc06ea4f00996fa066d719e418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.222450 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb698ff30fcc47d09000cc68d2c5225b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.229012 1345 raft_consensus.cc:493] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.229176 1345 raft_consensus.cc:515] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.229836 1345 leader_election.cc:290] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:13.231490 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b48bb260e854beb9ec61ce9c79b46c2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:13.231784 913 raft_consensus.cc:2468] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:13.232191 1206 tablet_bootstrap.cc:492] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:13.232353 1206 ts_tablet_manager.cc:1403] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.250s user 0.006s sys 0.014s
I20251214 20:52:13.232575 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_8b86c42f7826440982c88a8625559146"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:13.232725 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_75bf61ed45674e18a330a910511df0fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.233214 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b86c42f7826440982c88a8625559146 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.233247 1206 raft_consensus.cc:359] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:13.233415 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75bf61ed45674e18a330a910511df0fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.233434 1206 raft_consensus.cc:385] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.233546 1206 raft_consensus.cc:740] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.233752 1208 raft_consensus.cc:2243] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.235049 991 leader_election.cc:304] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:13.235427 1208 raft_consensus.cc:2272] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.236032 1345 raft_consensus.cc:2804] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:13.236152 1345 raft_consensus.cc:493] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:13.236228 1345 raft_consensus.cc:3060] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.238474 1202 tablet_bootstrap.cc:654] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.244979 1345 raft_consensus.cc:515] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.246572 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b48bb260e854beb9ec61ce9c79b46c2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:13.246881 913 raft_consensus.cc:3060] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.250200 913 raft_consensus.cc:2468] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:13.250842 991 leader_election.cc:304] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:13.251645 1345 leader_election.cc:290] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:13.251931 1345 raft_consensus.cc:2804] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:13.252110 1345 raft_consensus.cc:697] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:13.252489 1345 consensus_queue.cc:237] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.253023 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.258584 888 tablet_service.cc:1505] Processing CreateTablet for tablet bb698ff30fcc47d09000cc68d2c5225b (DEFAULT_TABLE table=default.Table_ab4c9e0bdaf74a1791ddad541ece2482 [id=8c96acfc06ea4f00996fa066d719e418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.259049 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb698ff30fcc47d09000cc68d2c5225b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.259723 1208 ts_tablet_manager.cc:1916] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.264364 1203 tablet_bootstrap.cc:492] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:13.264547 1203 ts_tablet_manager.cc:1403] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.000s
I20251214 20:52:13.267596 1203 raft_consensus.cc:359] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.267843 1203 raft_consensus.cc:385] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.267932 1203 raft_consensus.cc:740] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.270738 1203 consensus_queue.cc:260] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.274999 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 3c3962c33a294fa98f8bd485268aee71
I20251214 20:52:13.276150 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3962c33a294fa98f8bd485268aee71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.276279 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3962c33a294fa98f8bd485268aee71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.276970 1202 tablet_bootstrap.cc:492] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:13.276782 522 catalog_manager.cc:5654] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:13.277127 1202 ts_tablet_manager.cc:1403] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.003s
I20251214 20:52:13.277233 522 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 33a7f355b40f472883e3399209ef6762 (table default.Table_1c75ca10deb842669e2fea0f16ecc983 [id=5272550343144d4baaba07e97ebaa187]). Expected version 1 got 0
I20251214 20:52:13.277915 1202 raft_consensus.cc:359] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.278049 520 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet e8c02a2e4a2640b1b3a69f25a708684b
I20251214 20:52:13.278101 1202 raft_consensus.cc:385] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.278162 1202 raft_consensus.cc:740] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.278383 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_46ba3258cb8d46b2b0f28e6b98eb8603" } new_table_name: "default.Table_2695eb1d7e3a40c08c5a19eb6afce20f" modify_external_catalogs: true
I20251214 20:52:13.278437 1202 consensus_queue.cc:260] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.279970 1202 ts_tablet_manager.cc:1434] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20251214 20:52:13.279906 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_dbcc1409fc7e48cb804842a37eebaad2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:13.280264 1202 tablet_bootstrap.cc:492] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:13.280395 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbcc1409fc7e48cb804842a37eebaad2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:13.281559 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd88d1e95ee04ef596863e1ae3687de0 consensus state has no leader
I20251214 20:52:13.281654 1207 ts_tablet_manager.cc:1929] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.281836 1202 tablet_bootstrap.cc:654] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.281983 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_274bf0a7631e4c3c8144694db10a5de9" } new_table_name: "default.Table_8ccdf53e3ef149e1a70a20de6caedafa" modify_external_catalogs: true
I20251214 20:52:13.283460 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_b151b87a38944a7a89d0fe6ff10ae68b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.283922 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b151b87a38944a7a89d0fe6ff10ae68b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.285285 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.291903 1208 ts_tablet_manager.cc:1929] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.292086 1203 ts_tablet_manager.cc:1434] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20251214 20:52:13.292090 1207 log.cc:1199] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/d8947ebdc87341ad8b1a01f5dd246172
I20251214 20:52:13.292402 1203 tablet_bootstrap.cc:492] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:13.292590 1207 ts_tablet_manager.cc:1950] T d8947ebdc87341ad8b1a01f5dd246172 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:13.292604 1208 log.cc:1199] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/8c3b9ff3c6234507b76afd18f097b6e0
I20251214 20:52:13.293741 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 147) took 1138 ms. Trace:
I20251214 20:52:13.293856 1207 rpcz_store.cc:276] 1214 20:52:12.154535 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.154833 (+ 298us) service_pool.cc:225] Handling call
1214 20:52:12.905412 (+750579us) ts_tablet_manager.cc:1151] Deleting tablet d8947ebdc87341ad8b1a01f5dd246172
1214 20:52:12.905426 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:13.266061 (+360635us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.281471 (+ 15410us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.292943 (+ 11472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":728807}
I20251214 20:52:13.294039 1203 tablet_bootstrap.cc:654] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.294371 1206 consensus_queue.cc:260] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.294867 1208 ts_tablet_manager.cc:1950] T 8c3b9ff3c6234507b76afd18f097b6e0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:13.295960 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 8c3b9ff3c6234507b76afd18f097b6e0 (table default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]) successfully deleted
I20251214 20:52:13.299299 1208 tablet_replica.cc:333] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:13.299564 1208 raft_consensus.cc:2243] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:13.299993 1208 raft_consensus.cc:2272] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.304144 1208 ts_tablet_manager.cc:1916] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:13.309604 517 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2ae9dc8ead8403cb7215af8778872f0 consensus state has no leader
I20251214 20:52:13.309780 1415 raft_consensus.cc:697] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:13.313346 1206 ts_tablet_manager.cc:1434] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.080s user 0.003s sys 0.000s
I20251214 20:52:13.313849 1206 tablet_bootstrap.cc:492] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:13.316013 1206 tablet_bootstrap.cc:654] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.317049 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 149) took 1073 ms. Trace:
I20251214 20:52:13.317250 1207 rpcz_store.cc:276] 1214 20:52:12.220665 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.220734 (+ 69us) service_pool.cc:225] Handling call
1214 20:52:13.294207 (+1073473us) ts_tablet_manager.cc:1151] Deleting tablet b1df0dad2fb040a5a12e0401e00c87db
1214 20:52:13.294216 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:13.294334 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1073108}
I20251214 20:52:13.317471 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_5fbbdeeea9a14e17bf6f9d0cc4a56d98" } modify_external_catalogs: true
I20251214 20:52:13.317723 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_5fbbdeeea9a14e17bf6f9d0cc4a56d98" } modify_external_catalogs: true
I20251214 20:52:13.317868 1207 tablet_replica.cc:333] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:13.318089 1207 raft_consensus.cc:2243] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.318315 1207 raft_consensus.cc:2272] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.318969 1207 ts_tablet_manager.cc:1916] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:13.319743 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet cd88d1e95ee04ef596863e1ae3687de0 (table default.Table_bda8a86e13ba427e94e8d85a318281ad [id=2018af6df0b448bd8996691cb5a1f62c], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet cd88d1e95ee04ef596863e1ae3687de0 consensus state has no leader
I20251214 20:52:13.327067 1208 ts_tablet_manager.cc:1929] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:13.327661 1208 log.cc:1199] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/d8947ebdc87341ad8b1a01f5dd246172
I20251214 20:52:13.328294 1208 ts_tablet_manager.cc:1950] T d8947ebdc87341ad8b1a01f5dd246172 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:13.331965 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet d8947ebdc87341ad8b1a01f5dd246172 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]) successfully deleted
I20251214 20:52:13.332643 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d78784658d7d47e1b55c72607d6169b0" } new_table_name: "default.Table_a258f399b3e44e009cf7e014db9079aa" modify_external_catalogs: true
I20251214 20:52:13.333202 1208 tablet_replica.cc:333] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:13.333379 1208 raft_consensus.cc:2243] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.333559 1208 raft_consensus.cc:2272] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.334059 1208 ts_tablet_manager.cc:1916] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.335000 1202 tablet_bootstrap.cc:492] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:13.335230 1202 ts_tablet_manager.cc:1403] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.005s
I20251214 20:52:13.336179 1202 raft_consensus.cc:359] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.336478 1202 raft_consensus.cc:385] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.336625 1202 raft_consensus.cc:740] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.336911 1202 consensus_queue.cc:260] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.337466 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.348373 524 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_cd29381b062249718fbdc39a105f46a7 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 2 got 0
W20251214 20:52:13.348800 520 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0e805cc899347a083d4d549ad68e2f9 consensus state has no leader
I20251214 20:52:13.349718 1415 consensus_queue.cc:237] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.351994 888 tablet_service.cc:1552] Processing DeleteTablet for tablet dc442726ec0c456290ecf2892d0df957 with delete_type TABLET_DATA_DELETED (replaced by tablet 6bd6856b800247de896f16eee66f2827) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.353610 1202 ts_tablet_manager.cc:1434] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
I20251214 20:52:13.353922 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.355419 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33a7f355b40f472883e3399209ef6762" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:13.355691 1203 tablet_bootstrap.cc:492] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:13.355803 1056 raft_consensus.cc:2468] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:13.356362 1203 ts_tablet_manager.cc:1403] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.000s
I20251214 20:52:13.357179 1203 raft_consensus.cc:359] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.357375 1203 raft_consensus.cc:385] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.357442 1203 raft_consensus.cc:740] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.357688 1203 consensus_queue.cc:260] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.358233 1203 ts_tablet_manager.cc:1434] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:13.358563 1203 tablet_bootstrap.cc:492] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:13.358843 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b48bb260e854beb9ec61ce9c79b46c2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:13.359144 776 raft_consensus.cc:2468] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:13.359794 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b48bb260e854beb9ec61ce9c79b46c2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:13.359977 776 raft_consensus.cc:3060] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.361284 1202 tablet_bootstrap.cc:492] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:13.366019 776 raft_consensus.cc:2468] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:13.366901 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_fbb5b21626c142c8baac60a7f29d3c51" } modify_external_catalogs: true
I20251214 20:52:13.367229 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_fbb5b21626c142c8baac60a7f29d3c51" } modify_external_catalogs: true
W20251214 20:52:13.367601 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 023299180d074893847e84dcd9812751 consensus state has no leader
I20251214 20:52:13.374071 1202 tablet_bootstrap.cc:654] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.375500 1203 tablet_bootstrap.cc:654] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.377035 1433 mvcc.cc:204] Tried to move back new op lower bound from 7232493698743861248 to 7232493685663834112. Current Snapshot: MvccSnapshot[applied={T|T < 7232493698698346496}]
I20251214 20:52:13.377142 1207 ts_tablet_manager.cc:1929] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:13.378928 1207 log.cc:1199] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/44f51e11041841e98b9a15d380226e4e
I20251214 20:52:13.379971 1207 ts_tablet_manager.cc:1950] T 44f51e11041841e98b9a15d380226e4e P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:13.380578 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 52daf1204bf142e7b82852bdec6582fb
I20251214 20:52:13.381634 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 52daf1204bf142e7b82852bdec6582fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.381742 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 52daf1204bf142e7b82852bdec6582fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.384756 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 150) took 1163 ms. Trace:
I20251214 20:52:13.387557 1207 rpcz_store.cc:276] 1214 20:52:12.220945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.221146 (+ 201us) service_pool.cc:225] Handling call
1214 20:52:13.317763 (+1096617us) ts_tablet_manager.cc:1151] Deleting tablet 44f51e11041841e98b9a15d380226e4e
1214 20:52:13.317776 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:13.322297 (+ 4521us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.375465 (+ 53168us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.384700 (+ 9235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1096430}
I20251214 20:52:13.388499 1476 raft_consensus.cc:493] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.388971 1342 raft_consensus.cc:493] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.388900 1476 raft_consensus.cc:515] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.389145 1342 raft_consensus.cc:515] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.389827 1342 leader_election.cc:290] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:13.389878 1476 leader_election.cc:290] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.391139 1208 ts_tablet_manager.cc:1929] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.391176 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd0fa0dad974c98bf6f52c8cfe83035" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:13.391587 1208 log.cc:1199] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/47a67b1b9dc94e498cad91a72e655a75
W20251214 20:52:13.391960 851 leader_election.cc:343] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:13.392251 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd0fa0dad974c98bf6f52c8cfe83035" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:13.392504 1208 ts_tablet_manager.cc:1950] T 47a67b1b9dc94e498cad91a72e655a75 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
W20251214 20:52:13.392835 848 leader_election.cc:343] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:13.392963 848 leader_election.cc:304] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:13.393425 1342 raft_consensus.cc:2749] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:13.393908 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 334136us
I20251214 20:52:13.394703 755 tablet_service.cc:1505] Processing CreateTablet for tablet 07a8662a634a4befbd2d6251000d2059 (DEFAULT_TABLE table=default.Table_1a95326a5ce4436ca97f62d32966ec22 [id=6157817e96d642d09f9941ad7f7e82ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.395267 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a8662a634a4befbd2d6251000d2059. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.396620 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2acb51a56a60469e941100c99494a99b" } modify_external_catalogs: true
I20251214 20:52:13.396888 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2acb51a56a60469e941100c99494a99b" } modify_external_catalogs: true
I20251214 20:52:13.399284 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 47a67b1b9dc94e498cad91a72e655a75 (table default.Table_fa0e830600ba474bbbd6a68b9593d5f2 [id=16f2e0f0dbda4548871716a4a0f44dd4]) successfully deleted
W20251214 20:52:13.399992 518 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fcdd6bafa5c4aa6b59668cde523d010 consensus state has no leader
I20251214 20:52:13.400022 1208 tablet_replica.cc:333] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:13.400627 1208 raft_consensus.cc:2243] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.401388 1208 raft_consensus.cc:2272] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.402518 1208 ts_tablet_manager.cc:1916] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.394699 1207 tablet_replica.cc:333] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:13.408609 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 604b7c6fe3b24daca70711a1990af556
I20251214 20:52:13.409718 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7c6fe3b24daca70711a1990af556 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.413168 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b with delete_type TABLET_DATA_DELETED (replaced by tablet e8c02a2e4a2640b1b3a69f25a708684b) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.413985 742 tablet_service.cc:1505] Processing CreateTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 (DEFAULT_TABLE table=default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.414531 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bbb28259c1442ae94f4eff76c25c539. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.414480 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7c6fe3b24daca70711a1990af556 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.415729 1203 tablet_bootstrap.cc:492] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:13.415890 1203 ts_tablet_manager.cc:1403] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20251214 20:52:13.416671 1203 raft_consensus.cc:359] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.416859 1203 raft_consensus.cc:385] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.416946 1203 raft_consensus.cc:740] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.417181 1203 consensus_queue.cc:260] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.418030 1203 ts_tablet_manager.cc:1434] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:13.418399 1203 tablet_bootstrap.cc:492] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:13.420770 1203 tablet_bootstrap.cc:654] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.422279 755 tablet_service.cc:1505] Processing CreateTablet for tablet 604b7c6fe3b24daca70711a1990af556 (DEFAULT_TABLE table=default.Table_fbb5b21626c142c8baac60a7f29d3c51 [id=7cb78d704474480db14d212ee89166a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.422860 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604b7c6fe3b24daca70711a1990af556. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.425726 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 02e256d4537249a394b80dc36beefef4 (DEFAULT_TABLE table=default.Table_ecba341e009147b9b843be246ea2cd7c [id=ed8d16aa8d984c4198c1d1b4ae3fc729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.426292 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e256d4537249a394b80dc36beefef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.427918 1476 raft_consensus.cc:493] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.428107 1476 raft_consensus.cc:515] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.428750 1476 leader_election.cc:290] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.430243 888 tablet_service.cc:1505] Processing CreateTablet for tablet 02e256d4537249a394b80dc36beefef4 (DEFAULT_TABLE table=default.Table_ecba341e009147b9b843be246ea2cd7c [id=ed8d16aa8d984c4198c1d1b4ae3fc729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.430377 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c (DEFAULT_TABLE table=default.Table_caa5b0dabd9d4ea8a8b16c550407dc53 [id=4c909a51df124835bd806b8c6e53214e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.430677 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e256d4537249a394b80dc36beefef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.430753 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fcfcfe81fe24d369eea696efc61d21c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:13.437603 713 leader_election.cc:343] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 8c3b9ff3c6234507b76afd18f097b6e0
W20251214 20:52:13.438148 713 consensus_peers.cc:597] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 358d5c6b594147c8ba136b4b0732fae1. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:13.438441 713 leader_election.cc:304] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:13.438791 1476 raft_consensus.cc:2804] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 20:52:13.438879 1476 raft_consensus.cc:493] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:13.438969 1476 raft_consensus.cc:3060] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Advancing to term 2
I20251214 20:52:13.439251 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8e09c13caca4a1c9018387972c9f082" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:13.439472 1055 raft_consensus.cc:2468] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:13.442122 1476 raft_consensus.cc:515] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.443446 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "119c8cc0d6eb4d7f9a5d7fb3000c347b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:13.443709 1056 raft_consensus.cc:2468] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:13.444473 1476 leader_election.cc:290] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.444691 523 catalog_manager.cc:5654] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:13.445161 523 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_616b5b16d08b43eb8a04a21a352031fe [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 1 got 0
I20251214 20:52:13.208166 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.210544 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (DEFAULT_TABLE table=default.Table_d78784658d7d47e1b55c72607d6169b0 [id=ed335fb680bc441184e1682200ee68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.445503 1208 ts_tablet_manager.cc:1929] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.445647 1208 log.cc:1199] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/c17126f75b6241c5833269d20774c439
I20251214 20:52:13.445881 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fcdd6bafa5c4aa6b59668cde523d010. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.446233 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_7d6d418a1acb404f908e9ee299cef249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.446861 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d6d418a1acb404f908e9ee299cef249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.448184 1208 ts_tablet_manager.cc:1950] T c17126f75b6241c5833269d20774c439 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:13.448140 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_9ad469dea994477983bfe62eef37e334"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.448556 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ad469dea994477983bfe62eef37e334 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.451018 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet c17126f75b6241c5833269d20774c439 (table default.Table_834f1d323b2b4785a85f667e5c14e739 [id=587a162643cf4bc5a3b4c9fc0de83340]) successfully deleted
I20251214 20:52:13.452450 1208 tablet_replica.cc:333] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:13.453213 1208 raft_consensus.cc:2243] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.453881 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 472e4a5990e64d02b1d52dc6e86d7190
I20251214 20:52:13.454645 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.453940 1208 raft_consensus.cc:2272] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.456105 1208 ts_tablet_manager.cc:1916] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.456866 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.458737 1342 raft_consensus.cc:493] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.458882 1342 raft_consensus.cc:515] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.459530 1342 leader_election.cc:290] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:13.459841 1022 tablet_service.cc:1505] Processing CreateTablet for tablet e16fe751f10f4616a00d0af811998729 (DEFAULT_TABLE table=default.Table_181aab73cbf1461b85364fc893700db7 [id=8fd9be1206a74aa1921e0dade8be3774]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.459970 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af99e0bd8774a678575847a9ec2e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.460148 751 tablet_service.cc:1552] Processing DeleteTablet for tablet b652a8a80246443491551e76e68d2a8b with delete_type TABLET_DATA_DELETED (replaced by tablet 18e119bc44874aecb4e83c8d6040c8cd) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.460289 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16fe751f10f4616a00d0af811998729. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.460932 751 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3962c33a294fa98f8bd485268aee71 (DEFAULT_TABLE table=default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.461504 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3962c33a294fa98f8bd485268aee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.461647 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc442726ec0c456290ecf2892d0df957" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:13.462234 851 leader_election.cc:343] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:13.462670 888 tablet_service.cc:1505] Processing CreateTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c (DEFAULT_TABLE table=default.Table_caa5b0dabd9d4ea8a8b16c550407dc53 [id=4c909a51df124835bd806b8c6e53214e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.463042 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fcfcfe81fe24d369eea696efc61d21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.470340 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc442726ec0c456290ecf2892d0df957" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:13.470760 742 tablet_service.cc:1505] Processing CreateTablet for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (DEFAULT_TABLE table=default.Table_0191351f0cd945e489dc8d02dbe2c5be [id=3aaa67725e8442fdbae975274c8596c4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:13.471226 848 leader_election.cc:343] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:13.471392 848 leader_election.cc:304] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:13.471553 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f4760585baf4b4ab3b9b94b5a748a61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.472070 1342 raft_consensus.cc:2749] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:13.477869 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 2da66ec83e9140f7a605f0c86abff468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.478839 749 tablet_service.cc:1505] Processing CreateTablet for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (DEFAULT_TABLE table=default.Table_d78784658d7d47e1b55c72607d6169b0 [id=ed335fb680bc441184e1682200ee68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.479423 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fcdd6bafa5c4aa6b59668cde523d010. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.479337 1030 tablet_service.cc:1505] Processing CreateTablet for tablet b70579ddf96f4499b4e4f235e8f69d67 (DEFAULT_TABLE table=default.Table_3047cd5c72e34614924729dcc11eb7f4 [id=bdb017130eed4d76b635603560c91d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.479852 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70579ddf96f4499b4e4f235e8f69d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.481678 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet c17126f75b6241c5833269d20774c439 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.481949 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 47a67b1b9dc94e498cad91a72e655a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.488381 888 tablet_service.cc:1505] Processing CreateTablet for tablet e16fe751f10f4616a00d0af811998729 (DEFAULT_TABLE table=default.Table_181aab73cbf1461b85364fc893700db7 [id=8fd9be1206a74aa1921e0dade8be3774]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.488783 751 tablet_service.cc:1505] Processing CreateTablet for tablet 8d72f55d2d2440d993aa205d96c356f1 (DEFAULT_TABLE table=default.Table_2f947b9f382942e082ea4909657a8f34 [id=66d124c2a8dd4cc188a832eff1dbe9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.489187 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16fe751f10f4616a00d0af811998729. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.489373 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d72f55d2d2440d993aa205d96c356f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.492264 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_33ca298479304caaa7c59e715fd082b2" } new_table_name: "default.Table_a889ffcf345c4209b084b35515555d67" modify_external_catalogs: true
I20251214 20:52:13.492321 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_bda8a86e13ba427e94e8d85a318281ad" } modify_external_catalogs: true
I20251214 20:52:13.492555 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_bda8a86e13ba427e94e8d85a318281ad" } modify_external_catalogs: true
I20251214 20:52:13.492590 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4e46507057c94aa7a347491bf2e482ff" } modify_external_catalogs: true
I20251214 20:52:13.492817 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4e46507057c94aa7a347491bf2e482ff" } modify_external_catalogs: true
I20251214 20:52:13.492345 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_066df6ad9e814765a9866cc1dacd0997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.494542 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_066df6ad9e814765a9866cc1dacd0997 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.496631 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_c8d7238b9dc045a7bed9e721f7c022f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.497334 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8d7238b9dc045a7bed9e721f7c022f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.503743 1206 tablet_bootstrap.cc:492] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:13.504580 1206 ts_tablet_manager.cc:1403] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.191s user 0.009s sys 0.067s
I20251214 20:52:13.505398 1206 raft_consensus.cc:359] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.505657 1206 raft_consensus.cc:385] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.505775 1206 raft_consensus.cc:740] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.506038 1206 consensus_queue.cc:260] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.506574 1206 ts_tablet_manager.cc:1434] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:13.507026 1206 tablet_bootstrap.cc:492] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:13.504313 749 tablet_service.cc:1505] Processing CreateTablet for tablet bb698ff30fcc47d09000cc68d2c5225b (DEFAULT_TABLE table=default.Table_ab4c9e0bdaf74a1791ddad541ece2482 [id=8c96acfc06ea4f00996fa066d719e418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.507615 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb698ff30fcc47d09000cc68d2c5225b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.510986 1208 ts_tablet_manager.cc:1929] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.511533 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_3047cd5c72e34614924729dcc11eb7f4" } new_table_name: "default.Table_5879a88313cb488b8092323f94dfd3a7" modify_external_catalogs: true
I20251214 20:52:13.511773 1415 raft_consensus.cc:493] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.512008 1415 raft_consensus.cc:515] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.512717 1415 leader_election.cc:290] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.512791 1208 log.cc:1199] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/a8e09c13caca4a1c9018387972c9f082
I20251214 20:52:13.513449 1208 ts_tablet_manager.cc:1950] T a8e09c13caca4a1c9018387972c9f082 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:13.515317 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 52daf1204bf142e7b82852bdec6582fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.515590 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7c6fe3b24daca70711a1990af556 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.515745 747 tablet_service.cc:1505] Processing CreateTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c (DEFAULT_TABLE table=default.Table_caa5b0dabd9d4ea8a8b16c550407dc53 [id=4c909a51df124835bd806b8c6e53214e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.515998 754 tablet_service.cc:1505] Processing CreateTablet for tablet 02e256d4537249a394b80dc36beefef4 (DEFAULT_TABLE table=default.Table_ecba341e009147b9b843be246ea2cd7c [id=ed8d16aa8d984c4198c1d1b4ae3fc729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.516153 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fcfcfe81fe24d369eea696efc61d21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.516456 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e256d4537249a394b80dc36beefef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.518558 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet a8e09c13caca4a1c9018387972c9f082 (table default.Table_4a73de3f8186423abc9e9123ed3f1acb [id=397be7c374504dc5973e1cbf6b263084]) successfully deleted
I20251214 20:52:13.520040 888 tablet_service.cc:1505] Processing CreateTablet for tablet b70579ddf96f4499b4e4f235e8f69d67 (DEFAULT_TABLE table=default.Table_3047cd5c72e34614924729dcc11eb7f4 [id=bdb017130eed4d76b635603560c91d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.520602 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70579ddf96f4499b4e4f235e8f69d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.522796 1205 ts_tablet_manager.cc:1929] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.523023 1205 log.cc:1199] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/8140dad1fab142d98b930b53ae5a64c2
I20251214 20:52:13.523523 1205 ts_tablet_manager.cc:1950] T 8140dad1fab142d98b930b53ae5a64c2 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:13.524202 1208 tablet_replica.cc:333] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:13.525231 1206 tablet_bootstrap.cc:654] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.525280 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.525506 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3962c33a294fa98f8bd485268aee71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.526095 1208 raft_consensus.cc:2243] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:13.528059 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bd5c18696f54ea190c1ef84b93c4f63 consensus state has no leader
I20251214 20:52:13.528052 518 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet a0279992d1f2419ab1caae116a0391b2
I20251214 20:52:13.529767 880 tablet_service.cc:1552] Processing DeleteTablet for tablet a0279992d1f2419ab1caae116a0391b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.530128 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet a0279992d1f2419ab1caae116a0391b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.531116 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet d8947ebdc87341ad8b1a01f5dd246172 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]) successfully deleted
W20251214 20:52:13.531437 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b1df0dad2fb040a5a12e0401e00c87db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1df0dad2fb040a5a12e0401e00c87db already in progress: creating tablet
I20251214 20:52:13.531625 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 44f51e11041841e98b9a15d380226e4e (table default.Table_0191351f0cd945e489dc8d02dbe2c5be [id=3aaa67725e8442fdbae975274c8596c4]) successfully deleted
I20251214 20:52:13.532002 745 tablet_service.cc:1505] Processing CreateTablet for tablet e16fe751f10f4616a00d0af811998729 (DEFAULT_TABLE table=default.Table_181aab73cbf1461b85364fc893700db7 [id=8fd9be1206a74aa1921e0dade8be3774]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.534222 880 tablet_service.cc:1505] Processing CreateTablet for tablet 1e31615d1277412eb8bd80451f2c4366 (DEFAULT_TABLE table=default.Table_53273de8ee634ce9994c46fe98bba85e [id=cd14def069aa430994cd391ddab6c4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.534742 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e31615d1277412eb8bd80451f2c4366. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.540601 522 master_service.cc:954] ReplaceTablet: received request to replace tablet a34eb8fa3e34427495fa5985b7ba81bf from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:13.541961 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 9523d72fd91445539d67e1eac2a80afd from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:13.546631 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 2a7fc15882e2474bb6885070be13b1fa (DEFAULT_TABLE table=default.Table_727db60cbb7c435d986f772214b16166 [id=5095fbf5b9014d708cf7e7f48a4be93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.547274 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a7fc15882e2474bb6885070be13b1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.549495 1342 raft_consensus.cc:493] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.549690 1342 raft_consensus.cc:515] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.550418 1342 leader_election.cc:290] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:13.555491 1205 tablet_replica.cc:333] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:13.555747 1205 raft_consensus.cc:2243] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:13.555771 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 8140dad1fab142d98b930b53ae5a64c2 (table default.Table_d6e5176452664864914a6f76f32af5ac [id=3606c9fa9a3d4b55b86dddf293442a3c]) successfully deleted
W20251214 20:52:13.556118 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 84a0237e4ee34f94821d8647a400575b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84a0237e4ee34f94821d8647a400575b already in progress: creating tablet
W20251214 20:52:13.556310 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 12b5f4326aa0413eb303c132ba21decd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12b5f4326aa0413eb303c132ba21decd already in progress: creating tablet
W20251214 20:52:13.557333 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 81395us
I20251214 20:52:13.558890 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33a7f355b40f472883e3399209ef6762" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:13.559221 1056 raft_consensus.cc:3060] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Advancing to term 2
I20251214 20:52:13.560117 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c17126f75b6241c5833269d20774c439" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:13.560496 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_04cb0220358347e0b679db3dec97b700"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.561385 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04cb0220358347e0b679db3dec97b700 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.561663 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_501d1ae22d314c82914be72476d1ee0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.562171 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_501d1ae22d314c82914be72476d1ee0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.563246 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_4907bb640d114586b98e6fa0e514cb5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.563690 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4907bb640d114586b98e6fa0e514cb5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.565007 518 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_616b5b16d08b43eb8a04a21a352031fe [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 1 got 0
I20251214 20:52:13.565215 713 leader_election.cc:304] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:13.565810 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_384f071b4b4d411aa4fe7bcd8160a32a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.566246 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_384f071b4b4d411aa4fe7bcd8160a32a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.556023 1205 pending_rounds.cc:70] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:13.568149 756 tablet_service.cc:1505] Processing CreateTablet for tablet b70579ddf96f4499b4e4f235e8f69d67 (DEFAULT_TABLE table=default.Table_3047cd5c72e34614924729dcc11eb7f4 [id=bdb017130eed4d76b635603560c91d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.566267 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_0191351f0cd945e489dc8d02dbe2c5be" } new_table_name: "default.Table_f8f22fe9676e44c8931f7069ee213892" modify_external_catalogs: true
I20251214 20:52:13.569674 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed1d007f1ca24b7291262708ddfca3c3" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:13.570040 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16fe751f10f4616a00d0af811998729. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:13.570437 848 leader_election.cc:343] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:13.572335 1415 raft_consensus.cc:2804] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:13.572466 1415 raft_consensus.cc:493] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:13.572860 1415 raft_consensus.cc:3060] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.573964 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet cd88d1e95ee04ef596863e1ae3687de0
I20251214 20:52:13.574872 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_d5bf53032a5e4f659230b47347aa809d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:13.575984 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70579ddf96f4499b4e4f235e8f69d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.576635 880 tablet_service.cc:1552] Processing DeleteTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
W20251214 20:52:13.579363 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5bf53032a5e4f659230b47347aa809d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.580250 1203 tablet_bootstrap.cc:492] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:13.580443 1203 ts_tablet_manager.cc:1403] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.162s user 0.003s sys 0.000s
I20251214 20:52:13.581451 1203 raft_consensus.cc:359] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.582420 1055 raft_consensus.cc:2468] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:13.583046 1345 raft_consensus.cc:493] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.585413 1205 pending_rounds.cc:77] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493703241625600 op_type: NO_OP noop_request { }
I20251214 20:52:13.585820 880 tablet_service.cc:1505] Processing CreateTablet for tablet 2a7fc15882e2474bb6885070be13b1fa (DEFAULT_TABLE table=default.Table_727db60cbb7c435d986f772214b16166 [id=5095fbf5b9014d708cf7e7f48a4be93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.586328 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a7fc15882e2474bb6885070be13b1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.594354 1415 raft_consensus.cc:515] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.595232 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.595290 1345 raft_consensus.cc:515] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.596086 1345 leader_election.cc:290] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:13.596355 1056 raft_consensus.cc:2468] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 2.
I20251214 20:52:13.595028 1203 raft_consensus.cc:385] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.596745 1203 raft_consensus.cc:740] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.596966 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebedf28e17a741169e5490822a2d95df" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:13.597023 1203 consensus_queue.cc:260] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.597774 1203 ts_tablet_manager.cc:1434] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20251214 20:52:13.598163 1203 tablet_bootstrap.cc:492] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:13.598970 1207 raft_consensus.cc:2243] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:13.599521 1207 raft_consensus.cc:2272] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.600484 1203 tablet_bootstrap.cc:654] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.603214 1205 raft_consensus.cc:2889] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:13.603372 1205 raft_consensus.cc:2272] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.604313 1205 ts_tablet_manager.cc:1916] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.607450 1101 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.608755 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_0071d629c0444fa89511b0938f2e757e" } new_table_name: "default.Table_af66f4d0779b46a5a91d7d2490a845c9" modify_external_catalogs: true
I20251214 20:52:13.611749 1202 tablet_bootstrap.cc:492] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:13.612080 1202 ts_tablet_manager.cc:1403] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.251s user 0.003s sys 0.024s
I20251214 20:52:13.612707 1035 tablet_service.cc:1505] Processing CreateTablet for tablet a063083f54ad49d2a842fa080d44ed85 (DEFAULT_TABLE table=default.Table_15002dcbc05542cebdf323d46303384b [id=71344b71ec00402f8df0c119fd8704bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.612987 1202 raft_consensus.cc:359] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.613476 1202 raft_consensus.cc:385] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.613569 1202 raft_consensus.cc:740] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.613976 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a063083f54ad49d2a842fa080d44ed85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.614593 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed1d007f1ca24b7291262708ddfca3c3" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:13.615180 1202 consensus_queue.cc:260] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.615779 1202 ts_tablet_manager.cc:1434] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251214 20:52:13.616322 960 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.617372 1208 raft_consensus.cc:2272] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:13.620312 991 leader_election.cc:343] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:13.620410 851 leader_election.cc:343] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:13.620568 851 leader_election.cc:304] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:13.621950 1205 ts_tablet_manager.cc:1929] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:13.622092 1205 log.cc:1199] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/5ba2d34a6d2645bd97fe85fed952692b
I20251214 20:52:13.622639 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33a7f355b40f472883e3399209ef6762" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:13.623509 713 leader_election.cc:304] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:13.625588 1202 tablet_bootstrap.cc:492] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:13.625844 1342 raft_consensus.cc:2749] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:13.628424 523 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3fd0fa0dad974c98bf6f52c8cfe83035 (table default.Table_0173f37106d54db18ef3975bc6b0d185 [id=5c14810dd5364dcaa218b80c3756eadc]). Expected version 1 got 0
I20251214 20:52:13.631160 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "119c8cc0d6eb4d7f9a5d7fb3000c347b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:13.631502 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "33a7f355b40f472883e3399209ef6762" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:13.633546 1208 ts_tablet_manager.cc:1916] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.635412 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 376e5efede87410eb9f95f36bedebf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.636193 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_5de120dfaab44b84868db0d1f79e2cee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.636969 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5de120dfaab44b84868db0d1f79e2cee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:13.637945 524 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b70579ddf96f4499b4e4f235e8f69d67 consensus state has no leader
I20251214 20:52:13.638449 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d863e40d2a9e46649f9dc97f33a616db" } modify_external_catalogs: true
I20251214 20:52:13.638633 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d863e40d2a9e46649f9dc97f33a616db" } modify_external_catalogs: true
I20251214 20:52:13.639040 1415 leader_election.cc:290] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.642890 914 raft_consensus.cc:2468] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:13.643265 1476 raft_consensus.cc:2804] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:13.643277 880 tablet_service.cc:1505] Processing CreateTablet for tablet a063083f54ad49d2a842fa080d44ed85 (DEFAULT_TABLE table=default.Table_15002dcbc05542cebdf323d46303384b [id=71344b71ec00402f8df0c119fd8704bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.643450 1207 ts_tablet_manager.cc:1916] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.643816 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a063083f54ad49d2a842fa080d44ed85. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:13.644579 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 58093.1us
I20251214 20:52:13.646171 1476 raft_consensus.cc:493] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:13.646294 1476 raft_consensus.cc:3060] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.647325 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebedf28e17a741169e5490822a2d95df" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:13.650609 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ad4bec26fc854d67b2ec37f504851a01" } new_table_name: "default.Table_1ec14d4074ea4e058dcb928e5eddf5b2" modify_external_catalogs: true
I20251214 20:52:13.650811 1202 tablet_bootstrap.cc:654] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.656034 713 leader_election.cc:304] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:13.661684 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_aa507956e1b54af68730c20f7c10e842" } modify_external_catalogs: true
I20251214 20:52:13.661927 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_aa507956e1b54af68730c20f7c10e842" } modify_external_catalogs: true
I20251214 20:52:13.663987 889 tablet_service.cc:1505] Processing CreateTablet for tablet 9d89ee602c524c96a21c0a235af07481 (DEFAULT_TABLE table=default.Table_bb6020a9d77b42b5955933a449edc238 [id=8872bfac95c64cbba1ccb84d1d9f2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.664502 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d89ee602c524c96a21c0a235af07481. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.665583 1202 tablet_bootstrap.cc:492] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:13.665792 888 tablet_service.cc:1505] Processing CreateTablet for tablet ee7d709316b24b8eb223a838710c2979 (DEFAULT_TABLE table=default.Table_aa507956e1b54af68730c20f7c10e842 [id=f249932331aa4016806e482e8c7c4179]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.666240 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7d709316b24b8eb223a838710c2979. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.665799 1202 ts_tablet_manager.cc:1403] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.005s
I20251214 20:52:13.668666 1202 raft_consensus.cc:359] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.668864 1202 raft_consensus.cc:385] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.668953 1202 raft_consensus.cc:740] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.669306 1202 consensus_queue.cc:260] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.669921 1202 ts_tablet_manager.cc:1434] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:13.671277 1415 raft_consensus.cc:2804] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:13.674453 713 leader_election.cc:304] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:13.675197 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_847906e708bd41e9b9507efc8f52347f" } modify_external_catalogs: true
I20251214 20:52:13.675400 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_847906e708bd41e9b9507efc8f52347f" } modify_external_catalogs: true
I20251214 20:52:13.676700 1208 ts_tablet_manager.cc:1929] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.677700 1460 raft_consensus.cc:2804] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 20:52:13.677870 1460 raft_consensus.cc:697] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 2 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:13.678372 1460 consensus_queue.cc:237] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.679716 1415 raft_consensus.cc:493] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:13.679924 1476 raft_consensus.cc:515] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.680495 1476 leader_election.cc:290] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.680652 1208 log.cc:1199] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ebedf28e17a741169e5490822a2d95df
I20251214 20:52:13.681102 1208 ts_tablet_manager.cc:1950] T ebedf28e17a741169e5490822a2d95df P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:13.682032 1202 tablet_bootstrap.cc:492] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:13.686957 889 tablet_service.cc:1505] Processing CreateTablet for tablet ac64f4fcb6a943668584658df19f950c (DEFAULT_TABLE table=default.Table_5275181d46dc4a85b27fc0c45fa587fe [id=ab9838348eb24201bcb2866b16994965]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.687422 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac64f4fcb6a943668584658df19f950c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.688969 1342 raft_consensus.cc:493] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.689152 1342 raft_consensus.cc:515] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.689852 1342 leader_election.cc:290] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:13.691206 1415 raft_consensus.cc:3060] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.691674 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet a34eb8fa3e34427495fa5985b7ba81bf
I20251214 20:52:13.692179 522 catalog_manager.cc:6724] ReplaceTablet: tablet a34eb8fa3e34427495fa5985b7ba81bf deleted and replaced by tablet 6fef1d6add764fd28a8f4995b337ea71
I20251214 20:52:13.693374 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84a0237e4ee34f94821d8647a400575b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:13.693511 522 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 3fd0fa0dad974c98bf6f52c8cfe83035 (table default.Table_0173f37106d54db18ef3975bc6b0d185 [id=5c14810dd5364dcaa218b80c3756eadc]). Expected version 1 got 0
W20251214 20:52:13.694080 848 leader_election.cc:343] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:13.695708 882 tablet_service.cc:1552] Processing DeleteTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf with delete_type TABLET_DATA_DELETED (replaced by tablet 6fef1d6add764fd28a8f4995b337ea71) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.702924 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_f487575ab59448afa92cbbb86139dd53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.703745 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f487575ab59448afa92cbbb86139dd53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.703825 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 169) took 1027 ms. Trace:
I20251214 20:52:13.705173 822 heartbeater.cc:499] Master 127.0.104.60:35099 was elected leader, sending a full tablet report...
I20251214 20:52:13.704018 1208 rpcz_store.cc:276] 1214 20:52:12.669192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.669257 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:13.524121 (+854864us) ts_tablet_manager.cc:1151] Deleting tablet ebedf28e17a741169e5490822a2d95df
1214 20:52:13.524134 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:13.655987 (+131853us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.676532 (+ 20545us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.696791 (+ 20259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":854573}
I20251214 20:52:13.706415 1208 tablet_replica.cc:333] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:13.706434 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ebedf28e17a741169e5490822a2d95df (table default.Table_506ad42d4328442997401fa286d1874d [id=08c79b0f6de64d6397fbc34f9aa9fc96]) successfully deleted
I20251214 20:52:13.708858 1202 tablet_bootstrap.cc:654] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.709910 1208 raft_consensus.cc:2243] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.710136 1208 raft_consensus.cc:2272] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.707468 1314 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "63848a9fb78a4013a6f1657621272d75" instance_seqno: 1765745527881897) as {username='slave'} at 127.0.104.3:47651; Asking this server to re-register.
I20251214 20:52:13.711520 1342 consensus_queue.cc:1048] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:13.713734 1208 ts_tablet_manager.cc:1916] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.633090 1030 tablet_service.cc:1505] Processing CreateTablet for tablet ee7d709316b24b8eb223a838710c2979 (DEFAULT_TABLE table=default.Table_aa507956e1b54af68730c20f7c10e842 [id=f249932331aa4016806e482e8c7c4179]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.723181 745 tablet_service.cc:1552] Processing DeleteTablet for tablet a0279992d1f2419ab1caae116a0391b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.723536 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7d709316b24b8eb223a838710c2979. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:13.724045 518 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f4760585baf4b4ab3b9b94b5a748a61 consensus state has no leader
I20251214 20:52:13.724046 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_618d0aa0d8bf4389a544496b923a9db8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:13.725431 1415 raft_consensus.cc:515] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:13.725664 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_618d0aa0d8bf4389a544496b923a9db8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.645440 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8e09c13caca4a1c9018387972c9f082" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:13.655656 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 9d89ee602c524c96a21c0a235af07481 (DEFAULT_TABLE table=default.Table_bb6020a9d77b42b5955933a449edc238 [id=8872bfac95c64cbba1ccb84d1d9f2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.727308 1056 raft_consensus.cc:3060] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.727545 745 tablet_service.cc:1505] Processing CreateTablet for tablet 1e31615d1277412eb8bd80451f2c4366 (DEFAULT_TABLE table=default.Table_53273de8ee634ce9994c46fe98bba85e [id=cd14def069aa430994cd391ddab6c4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.727708 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d89ee602c524c96a21c0a235af07481. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.727929 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e31615d1277412eb8bd80451f2c4366. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.730225 1206 tablet_bootstrap.cc:492] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:13.730190 1056 raft_consensus.cc:2468] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:13.730352 1206 ts_tablet_manager.cc:1403] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.223s user 0.002s sys 0.013s
I20251214 20:52:13.731178 1206 raft_consensus.cc:359] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.731343 1206 raft_consensus.cc:385] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.731427 1206 raft_consensus.cc:740] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.731624 1206 consensus_queue.cc:260] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.655624 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 1e31615d1277412eb8bd80451f2c4366 (DEFAULT_TABLE table=default.Table_53273de8ee634ce9994c46fe98bba85e [id=cd14def069aa430994cd391ddab6c4dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.667580 1029 tablet_service.cc:1505] Processing CreateTablet for tablet ac64f4fcb6a943668584658df19f950c (DEFAULT_TABLE table=default.Table_5275181d46dc4a85b27fc0c45fa587fe [id=ab9838348eb24201bcb2866b16994965]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.733033 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac64f4fcb6a943668584658df19f950c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.675156 989 leader_election.cc:304] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:13.684056 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "119c8cc0d6eb4d7f9a5d7fb3000c347b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:13.735306 1055 raft_consensus.cc:3060] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:13.735476 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 60400.1us
I20251214 20:52:13.690579 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84a0237e4ee34f94821d8647a400575b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:13.698148 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf with delete_type TABLET_DATA_DELETED (replaced by tablet 6fef1d6add764fd28a8f4995b337ea71) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.707718 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (DEFAULT_TABLE table=default.Table_9e89276f2f2b4a7981f91242bdb24a87 [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.714939 1101 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:13.722025 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 35a529702f924a5c9bf53c3c668c6094 (DEFAULT_TABLE table=default.Table_73d965cabee84dd3865221877063447c [id=b0f347532cc84fcca5892f5a2443321c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.737325 1207 ts_tablet_manager.cc:1929] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:13.737656 1415 leader_election.cc:290] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.737748 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_15002dcbc05542cebdf323d46303384b" } modify_external_catalogs: true
I20251214 20:52:13.737977 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_15002dcbc05542cebdf323d46303384b" } modify_external_catalogs: true
I20251214 20:52:13.738520 880 tablet_service.cc:1505] Processing CreateTablet for tablet 35a529702f924a5c9bf53c3c668c6094 (DEFAULT_TABLE table=default.Table_73d965cabee84dd3865221877063447c [id=b0f347532cc84fcca5892f5a2443321c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.739055 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a529702f924a5c9bf53c3c668c6094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.739375 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_93388fdcfc2146b1b4e0f3554a12f10f" } new_table_name: "default.Table_d7519f28730149dc956626fb163db031" modify_external_catalogs: true
I20251214 20:52:13.739786 889 tablet_service.cc:1505] Processing CreateTablet for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (DEFAULT_TABLE table=default.Table_9e89276f2f2b4a7981f91242bdb24a87 [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.740839 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65b2fe0de22646e4b77fbc6a289ff4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.744216 1345 raft_consensus.cc:2749] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:13.744503 713 leader_election.cc:304] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:13.744740 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e31615d1277412eb8bd80451f2c4366. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.745536 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a8e09c13caca4a1c9018387972c9f082 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
W20251214 20:52:13.745903 851 leader_election.cc:343] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:13.746073 851 leader_election.cc:304] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:13.746522 1342 raft_consensus.cc:2749] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:13.747895 1206 ts_tablet_manager.cc:1434] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
W20251214 20:52:13.748232 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf37c50e511f454ca4887c7f6ca1a746 consensus state has no leader
I20251214 20:52:13.748389 1206 tablet_bootstrap.cc:492] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:13.749171 1207 log.cc:1199] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/358d5c6b594147c8ba136b4b0732fae1
I20251214 20:52:13.750603 751 tablet_service.cc:1552] Processing DeleteTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.750952 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a529702f924a5c9bf53c3c668c6094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.751307 1460 raft_consensus.cc:2804] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:13.751994 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65b2fe0de22646e4b77fbc6a289ff4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.753805 882 tablet_service.cc:1505] Processing CreateTablet for tablet 762068d560ff47918dd9fd8791083433 (DEFAULT_TABLE table=default.Table_431bf74dfb32400bb0b2ef098c18e8df [id=2fcb81bfc6904fdd924de2494aba2e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.754518 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762068d560ff47918dd9fd8791083433. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.754739 751 tablet_service.cc:1505] Processing CreateTablet for tablet 2a7fc15882e2474bb6885070be13b1fa (DEFAULT_TABLE table=default.Table_727db60cbb7c435d986f772214b16166 [id=5095fbf5b9014d708cf7e7f48a4be93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.755203 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a7fc15882e2474bb6885070be13b1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.755221 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 762068d560ff47918dd9fd8791083433 (DEFAULT_TABLE table=default.Table_431bf74dfb32400bb0b2ef098c18e8df [id=2fcb81bfc6904fdd924de2494aba2e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.758445 1460 raft_consensus.cc:697] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:13.758736 1460 consensus_queue.cc:237] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.759011 1055 raft_consensus.cc:2468] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:13.759374 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762068d560ff47918dd9fd8791083433. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.760183 1207 ts_tablet_manager.cc:1950] T 358d5c6b594147c8ba136b4b0732fae1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:13.760574 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 151) took 1539 ms. Trace:
I20251214 20:52:13.760689 1207 rpcz_store.cc:276] 1214 20:52:12.221248 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.221352 (+ 104us) service_pool.cc:225] Handling call
1214 20:52:13.390873 (+1169521us) ts_tablet_manager.cc:1151] Deleting tablet 358d5c6b594147c8ba136b4b0732fae1
1214 20:52:13.394601 (+ 3728us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x61800000c518. stack: 00007f95284c1ee6 00007f95284c1ca0 00007f952749d3ad 0000000000984f94 0000000000984f5d 000000000098492a 00007f95284aa81d 00007f95284ab88e 00007f953c10abdd 00007f953c089193 00007f95382b7b06 00007f95382b8e99 00007f95382ec744 00007f95382cd240 00007f95382ccfcd 0000000000a8970c
1214 20:52:13.394611 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:13.599221 (+204610us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x6160000d0c54. stack: 00007f95284c1ee6 00007f95284c1ca0 00007f952749d3ad 0000000000984f94 0000000000984f5d 000000000098492a 00007f95343ab1e0 00007f95356f90b3 00007f95382b9757 00007f95382ec744 00007f95382cd240 00007f95382ccfcd 0000000000a8970c 00007f952850820f 00007f952850b52b 00007f952850b3a8
1214 20:52:13.688843 (+ 89622us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.728568 (+ 39725us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.760489 (+ 31921us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":369300096,"tablet-delete.queue_time_us":1169372}
I20251214 20:52:13.760851 713 leader_election.cc:304] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:13.761209 1207 tablet_replica.cc:333] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:13.761482 1476 raft_consensus.cc:2804] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:13.761636 1476 raft_consensus.cc:697] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:13.762670 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c17126f75b6241c5833269d20774c439" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:13.762948 1056 raft_consensus.cc:3060] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.764627 1476 consensus_queue.cc:237] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.767431 1314 ts_manager.cc:194] Registered new tserver with Master: 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.774917 526 catalog_manager.cc:5654] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 reported cstate change: term changed from 1 to 2, leader changed from a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2) to 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1). New cstate: current_term: 2 leader_uuid: "074399864fb2470fa9e2ee87c1fe2212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:13.775478 526 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 33a7f355b40f472883e3399209ef6762 (table default.Table_1c75ca10deb842669e2fea0f16ecc983 [id=5272550343144d4baaba07e97ebaa187]). Expected version 1 got 0
I20251214 20:52:13.780097 747 tablet_service.cc:1505] Processing CreateTablet for tablet ac64f4fcb6a943668584658df19f950c (DEFAULT_TABLE table=default.Table_5275181d46dc4a85b27fc0c45fa587fe [id=ab9838348eb24201bcb2866b16994965]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.782541 1056 raft_consensus.cc:2468] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:13.783521 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac64f4fcb6a943668584658df19f950c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.783042 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2695eb1d7e3a40c08c5a19eb6afce20f" } modify_external_catalogs: true
I20251214 20:52:13.784138 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2695eb1d7e3a40c08c5a19eb6afce20f" } modify_external_catalogs: true
I20251214 20:52:13.784438 713 leader_election.cc:304] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:13.784888 1465 raft_consensus.cc:2804] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:13.785024 1465 raft_consensus.cc:697] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:13.785344 1465 consensus_queue.cc:237] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.787295 742 tablet_service.cc:1505] Processing CreateTablet for tablet 9d89ee602c524c96a21c0a235af07481 (DEFAULT_TABLE table=default.Table_bb6020a9d77b42b5955933a449edc238 [id=8872bfac95c64cbba1ccb84d1d9f2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.787326 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (DEFAULT_TABLE table=default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.787799 755 tablet_service.cc:1505] Processing CreateTablet for tablet a063083f54ad49d2a842fa080d44ed85 (DEFAULT_TABLE table=default.Table_15002dcbc05542cebdf323d46303384b [id=71344b71ec00402f8df0c119fd8704bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.789916 882 tablet_service.cc:1505] Processing CreateTablet for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (DEFAULT_TABLE table=default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.790437 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd5c18696f54ea190c1ef84b93c4f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.790551 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd5c18696f54ea190c1ef84b93c4f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.795519 756 tablet_service.cc:1505] Processing CreateTablet for tablet ee7d709316b24b8eb223a838710c2979 (DEFAULT_TABLE table=default.Table_aa507956e1b54af68730c20f7c10e842 [id=f249932331aa4016806e482e8c7c4179]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.796105 1202 tablet_bootstrap.cc:492] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:13.796350 1202 ts_tablet_manager.cc:1403] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.114s user 0.002s sys 0.002s
I20251214 20:52:13.797256 1202 raft_consensus.cc:359] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.797472 1202 raft_consensus.cc:385] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.797556 1202 raft_consensus.cc:740] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.797858 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d89ee602c524c96a21c0a235af07481. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.798198 1202 consensus_queue.cc:260] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.804056 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7d709316b24b8eb223a838710c2979. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.804822 1207 raft_consensus.cc:2243] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:13.805392 1205 ts_tablet_manager.cc:1950] T 5ba2d34a6d2645bd97fe85fed952692b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:13.807981 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 172) took 1011 ms. Trace:
I20251214 20:52:13.808214 1205 rpcz_store.cc:276] 1214 20:52:12.794908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.799716 (+ 4808us) service_pool.cc:225] Handling call
1214 20:52:13.555385 (+755669us) ts_tablet_manager.cc:1151] Deleting tablet 5ba2d34a6d2645bd97fe85fed952692b
1214 20:52:13.555391 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:13.615442 (+ 60051us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.621821 (+ 6379us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.806405 (+184584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":755489}
I20251214 20:52:13.810973 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet d6c0f445cd7e44769d79ad5566fd8d97
I20251214 20:52:13.812245 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a063083f54ad49d2a842fa080d44ed85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.815512 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 5ba2d34a6d2645bd97fe85fed952692b (table default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90]) successfully deleted
I20251214 20:52:13.812357 1203 tablet_bootstrap.cc:492] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:13.815999 889 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.817813 889 tablet_service.cc:1505] Processing CreateTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b (DEFAULT_TABLE table=default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.818288 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da9deeedce0421c9aeaebd71ba11b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.818696 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b (DEFAULT_TABLE table=default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.819164 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da9deeedce0421c9aeaebd71ba11b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.821951 1203 ts_tablet_manager.cc:1403] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.223s user 0.003s sys 0.004s
I20251214 20:52:13.824227 1203 raft_consensus.cc:359] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.824498 1203 raft_consensus.cc:385] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.824623 1203 raft_consensus.cc:740] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.824914 1203 consensus_queue.cc:260] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.826532 1203 ts_tablet_manager.cc:1434] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:13.815948 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:13.828642 521 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 569a02142c71427d91868b02695ae12b consensus state has no leader
I20251214 20:52:13.833487 1202 ts_tablet_manager.cc:1434] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20251214 20:52:13.838820 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff (DEFAULT_TABLE table=default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.838975 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3736aabecd40417f9cb502ac612bacee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:13.839172 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_e532e5fa17f2476a90df03eb6449a067"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.839773 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e532e5fa17f2476a90df03eb6449a067 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.840440 1314 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" instance_seqno: 1765745527287656) as {username='slave'} at 127.0.104.2:45379; Asking this server to re-register.
W20251214 20:52:13.841440 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3736aabecd40417f9cb502ac612bacee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.841627 1499 catalog_manager.cc:1269] Loaded cluster ID: c3f04138608d4adc935bb66e1a72f2bb
I20251214 20:52:13.841701 1499 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: loading cluster ID for follower catalog manager: success
I20251214 20:52:13.842581 960 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:13.844197 1202 tablet_bootstrap.cc:492] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:13.845008 1314 ts_manager.cc:194] Registered new tserver with Master: a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:13.848493 1207 pending_rounds.cc:70] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:13.850466 745 tablet_service.cc:1505] Processing CreateTablet for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (DEFAULT_TABLE table=default.Table_9e89276f2f2b4a7981f91242bdb24a87 [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.851485 889 tablet_service.cc:1505] Processing CreateTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff (DEFAULT_TABLE table=default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.852049 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e5e220d083243eb8aa4328f366cd7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.854511 1345 raft_consensus.cc:493] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.854730 1345 raft_consensus.cc:515] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.855525 1345 leader_election.cc:290] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:13.857478 989 leader_election.cc:343] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 47a67b1b9dc94e498cad91a72e655a75
W20251214 20:52:13.857668 991 leader_election.cc:343] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 47a67b1b9dc94e498cad91a72e655a75
I20251214 20:52:13.857865 991 leader_election.cc:304] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:13.859874 1345 raft_consensus.cc:2749] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:13.816251 1205 tablet_replica.cc:333] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:13.863492 1205 raft_consensus.cc:2243] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.863845 1205 raft_consensus.cc:2272] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.865022 1205 ts_tablet_manager.cc:1916] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.865592 1203 tablet_bootstrap.cc:492] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:13.869097 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e5e220d083243eb8aa4328f366cd7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.873363 519 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet ee7d709316b24b8eb223a838710c2979
I20251214 20:52:13.875344 1203 tablet_bootstrap.cc:654] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.851033 747 tablet_service.cc:1552] Processing DeleteTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf with delete_type TABLET_DATA_DELETED (replaced by tablet 6fef1d6add764fd28a8f4995b337ea71) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.851269 1207 pending_rounds.cc:77] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493704606281728 op_type: NO_OP noop_request { }
I20251214 20:52:13.852223 754 tablet_service.cc:1505] Processing CreateTablet for tablet 762068d560ff47918dd9fd8791083433 (DEFAULT_TABLE table=default.Table_431bf74dfb32400bb0b2ef098c18e8df [id=2fcb81bfc6904fdd924de2494aba2e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.852855 756 tablet_service.cc:1505] Processing CreateTablet for tablet 35a529702f924a5c9bf53c3c668c6094 (DEFAULT_TABLE table=default.Table_73d965cabee84dd3865221877063447c [id=b0f347532cc84fcca5892f5a2443321c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.855002 749 tablet_service.cc:1505] Processing CreateTablet for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (DEFAULT_TABLE table=default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.856220 1206 tablet_bootstrap.cc:654] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.869866 755 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.870239 742 tablet_service.cc:1505] Processing CreateTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b (DEFAULT_TABLE table=default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.870582 751 tablet_service.cc:1505] Processing CreateTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff (DEFAULT_TABLE table=default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.877061 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762068d560ff47918dd9fd8791083433. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.873142 748 tablet_service.cc:1505] Processing CreateTablet for tablet 18e119bc44874aecb4e83c8d6040c8cd (DEFAULT_TABLE table=default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.881304 1207 raft_consensus.cc:2889] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:13.881546 1207 raft_consensus.cc:2272] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.882849 519 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 569a02142c71427d91868b02695ae12b (table default.Table_1ec14d4074ea4e058dcb928e5eddf5b2 [id=17c89c03be10405f834504b803e594c7]). Expected version 1 got 0
I20251214 20:52:13.884699 1207 ts_tablet_manager.cc:1916] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.884965 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_cd29381b062249718fbdc39a105f46a7" } new_table_name: "default.Table_c2b912ceb2e64a64a51f04e3cf5d8101" modify_external_catalogs: true
I20251214 20:52:13.886664 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "119c8cc0d6eb4d7f9a5d7fb3000c347b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:13.886947 914 raft_consensus.cc:3060] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.888211 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 358d5c6b594147c8ba136b4b0732fae1 (table default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]) successfully deleted
I20251214 20:52:13.889828 914 raft_consensus.cc:2468] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:13.890328 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet ee7d709316b24b8eb223a838710c2979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.890574 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet ed1d007f1ca24b7291262708ddfca3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:13.890913 712 leader_election.cc:343] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 8c3b9ff3c6234507b76afd18f097b6e0
I20251214 20:52:13.891191 712 leader_election.cc:304] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:13.891579 1460 raft_consensus.cc:2749] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:13.891523 888 tablet_service.cc:1505] Processing CreateTablet for tablet 18e119bc44874aecb4e83c8d6040c8cd (DEFAULT_TABLE table=default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.891956 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e119bc44874aecb4e83c8d6040c8cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:13.892560 712 leader_election.cc:343] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: c17126f75b6241c5833269d20774c439
W20251214 20:52:13.892969 712 leader_election.cc:343] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: a8e09c13caca4a1c9018387972c9f082
W20251214 20:52:13.893687 712 leader_election.cc:343] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: a8e09c13caca4a1c9018387972c9f082
W20251214 20:52:13.894083 712 consensus_peers.cc:597] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebedf28e17a741169e5490822a2d95df. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:13.894378 712 leader_election.cc:343] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: c17126f75b6241c5833269d20774c439
I20251214 20:52:13.895287 755 tablet_service.cc:1552] Processing DeleteTablet for tablet ee7d709316b24b8eb223a838710c2979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.895504 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd5c18696f54ea190c1ef84b93c4f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.897038 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 18e119bc44874aecb4e83c8d6040c8cd (DEFAULT_TABLE table=default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:13.897483 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e119bc44874aecb4e83c8d6040c8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.899446 1345 raft_consensus.cc:493] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.899688 1345 raft_consensus.cc:515] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:13.900490 1345 leader_election.cc:290] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:13.901588 525 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 40a9a6192c744282ab5672cbe15fd299
I20251214 20:52:13.903800 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2b66900d6ea42aa849c24acbbe23c40" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:13.904165 914 raft_consensus.cc:2393] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63848a9fb78a4013a6f1657621272d75 in current term 1: Already voted for candidate 074399864fb2470fa9e2ee87c1fe2212 in this term.
I20251214 20:52:13.904381 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 40a9a6192c744282ab5672cbe15fd299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.906255 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 40a9a6192c744282ab5672cbe15fd299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.905706 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 40a9a6192c744282ab5672cbe15fd299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:13.907584 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2b66900d6ea42aa849c24acbbe23c40" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:13.908447 989 leader_election.cc:343] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: Tablet not RUNNING: STOPPED
I20251214 20:52:13.908668 989 leader_election.cc:304] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:13.908568 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_eb3e055643ea47539ad93faf71ed8cf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:13.909024 1345 raft_consensus.cc:3060] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:13.909324 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb3e055643ea47539ad93faf71ed8cf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.909755 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a529702f924a5c9bf53c3c668c6094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.910076 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e119bc44874aecb4e83c8d6040c8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.911819 1345 raft_consensus.cc:2749] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:13.913532 1056 raft_consensus.cc:3060] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.917825 1460 consensus_queue.cc:1048] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:13.922358 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e5e220d083243eb8aa4328f366cd7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.927537 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da9deeedce0421c9aeaebd71ba11b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.927770 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_585da05fa85a4320a84e53b8249d4aaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:13.928524 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_585da05fa85a4320a84e53b8249d4aaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.937642 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65b2fe0de22646e4b77fbc6a289ff4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:13.939630 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 24ff58f321124d12b5330dca185d426e from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:13.946682 1208 ts_tablet_manager.cc:1929] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.948213 880 tablet_service.cc:1552] Processing DeleteTablet for tablet ee7d709316b24b8eb223a838710c2979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.960659 1476 raft_consensus.cc:493] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:13.960933 1476 raft_consensus.cc:515] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.961922 1476 leader_election.cc:290] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.962234 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd0fa0dad974c98bf6f52c8cfe83035" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:13.962499 1055 raft_consensus.cc:2468] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:13.962946 713 leader_election.cc:304] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:13.963281 1476 raft_consensus.cc:2804] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:13.963379 1476 raft_consensus.cc:493] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:13.963481 1476 raft_consensus.cc:3060] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.964248 1203 tablet_bootstrap.cc:492] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:13.964426 1203 ts_tablet_manager.cc:1403] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.099s user 0.000s sys 0.012s
I20251214 20:52:13.965273 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet a063083f54ad49d2a842fa080d44ed85
I20251214 20:52:13.966181 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_9db4b4a52f48436eaa4d06ee1e411237"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:13.966426 1203 raft_consensus.cc:359] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.966696 1203 raft_consensus.cc:385] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:13.966769 1203 raft_consensus.cc:740] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:13.966758 1476 raft_consensus.cc:515] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
W20251214 20:52:13.966979 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9db4b4a52f48436eaa4d06ee1e411237 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.967979 1205 ts_tablet_manager.cc:1929] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:13.968125 1205 log.cc:1199] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ecb69e385bf34709b5d954de4b660087
I20251214 20:52:13.968530 1205 ts_tablet_manager.cc:1950] T ecb69e385bf34709b5d954de4b660087 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:13.970101 1476 leader_election.cc:290] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:13.968905 1203 consensus_queue.cc:260] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.971328 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a063083f54ad49d2a842fa080d44ed85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:13.971773 1203 ts_tablet_manager.cc:1434] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20251214 20:52:13.969910 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd0fa0dad974c98bf6f52c8cfe83035" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:13.972021 1055 raft_consensus.cc:3060] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:13.973203 851 consensus_peers.cc:597] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8140dad1fab142d98b930b53ae5a64c2. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:13.974687 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd0fa0dad974c98bf6f52c8cfe83035" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:13.974929 914 raft_consensus.cc:2468] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:13.975168 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fd0fa0dad974c98bf6f52c8cfe83035" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:13.976478 1203 tablet_bootstrap.cc:492] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:13.977093 1208 log.cc:1199] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/118dd44151a34ba3ae58f9fe1d3790d4
I20251214 20:52:13.977672 915 raft_consensus.cc:3060] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:13.977742 1202 tablet_bootstrap.cc:654] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.980473 880 tablet_service.cc:1552] Processing DeleteTablet for tablet a063083f54ad49d2a842fa080d44ed85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:13.982074 1055 raft_consensus.cc:2468] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:13.983400 1208 ts_tablet_manager.cc:1950] T 118dd44151a34ba3ae58f9fe1d3790d4 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:13.983675 713 leader_election.cc:304] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:13.985002 915 raft_consensus.cc:2468] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:13.985400 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 170) took 1309 ms. Trace:
I20251214 20:52:13.986336 1476 raft_consensus.cc:2804] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:13.986579 1476 raft_consensus.cc:697] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:13.986896 1476 consensus_queue.cc:237] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:13.990978 1208 rpcz_store.cc:276] 1214 20:52:12.676021 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.676075 (+ 54us) service_pool.cc:225] Handling call
1214 20:52:13.706302 (+1030227us) ts_tablet_manager.cc:1151] Deleting tablet 118dd44151a34ba3ae58f9fe1d3790d4
1214 20:52:13.706315 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:13.814433 (+108118us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.946510 (+132077us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.985340 (+ 38830us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029976}
I20251214 20:52:13.983004 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 173) took 1187 ms. Trace:
I20251214 20:52:13.991739 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 171) took 1297 ms. Trace:
I20251214 20:52:13.991811 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 118dd44151a34ba3ae58f9fe1d3790d4 (table default.Table_e8acf2f630434ce3b7f06a4a01e4a2a3 [id=ea94fa24050345f98516132203c9704c]) successfully deleted
I20251214 20:52:13.992755 1205 rpcz_store.cc:276] 1214 20:52:12.795026 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.795374 (+ 348us) service_pool.cc:225] Handling call
1214 20:52:13.816102 (+1020728us) ts_tablet_manager.cc:1151] Deleting tablet ecb69e385bf34709b5d954de4b660087
1214 20:52:13.816114 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:13.928573 (+112459us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.967803 (+ 39230us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.982959 (+ 15156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1015981}
W20251214 20:52:13.993160 518 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1389716f5ce44f588c21a34c7a7d4c8 has no consensus state
I20251214 20:52:13.993368 1205 tablet_replica.cc:333] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:13.993532 518 master_service.cc:954] ReplaceTablet: received request to replace tablet a34eb8fa3e34427495fa5985b7ba81bf from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:13.994001 518 master_service.cc:954] ReplaceTablet: received request to replace tablet a0279992d1f2419ab1caae116a0391b2 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:13.994215 1203 tablet_bootstrap.cc:654] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:13.994549 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_b92e50faaffb45c0a3e1c7a94166f985"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:13.995250 1205 raft_consensus.cc:2243] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:13.995463 1205 raft_consensus.cc:2272] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:13.995446 1208 rpcz_store.cc:276] 1214 20:52:12.694018 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.694068 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:13.991553 (+1297485us) ts_tablet_manager.cc:1151] Deleting tablet c25832da5c1644329e18653000f3330e
1214 20:52:13.991564 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:13.991704 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1297206}
I20251214 20:52:13.996342 1205 ts_tablet_manager.cc:1916] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:13.996918 1208 tablet_replica.cc:333] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
W20251214 20:52:13.997989 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b92e50faaffb45c0a3e1c7a94166f985 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:13.998365 1208 raft_consensus.cc:2243] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:13.998812 1208 raft_consensus.cc:2272] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.001581 1207 ts_tablet_manager.cc:1929] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:14.002061 1207 log.cc:1199] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/f2b66900d6ea42aa849c24acbbe23c40
I20251214 20:52:14.004580 1207 ts_tablet_manager.cc:1950] T f2b66900d6ea42aa849c24acbbe23c40 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:14.008347 1206 tablet_bootstrap.cc:492] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:14.008621 1206 ts_tablet_manager.cc:1403] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.260s user 0.006s sys 0.004s
I20251214 20:52:14.009192 1202 tablet_bootstrap.cc:492] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:14.009400 1202 ts_tablet_manager.cc:1403] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.165s user 0.003s sys 0.003s
I20251214 20:52:14.009754 1206 raft_consensus.cc:359] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.010106 1206 raft_consensus.cc:385] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.010234 1206 raft_consensus.cc:740] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.010349 1202 raft_consensus.cc:359] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.010571 1202 raft_consensus.cc:385] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.010615 1499 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: acquiring CA information for follower catalog manager: success
I20251214 20:52:14.010663 1202 raft_consensus.cc:740] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.010702 1206 consensus_queue.cc:260] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.011828 1206 ts_tablet_manager.cc:1434] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:14.012365 1206 tablet_bootstrap.cc:492] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:14.012804 1205 ts_tablet_manager.cc:1929] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:14.012992 1205 log.cc:1199] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/d8947ebdc87341ad8b1a01f5dd246172
I20251214 20:52:14.013368 1205 ts_tablet_manager.cc:1950] T d8947ebdc87341ad8b1a01f5dd246172 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:14.014284 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 176) took 1205 ms. Trace:
I20251214 20:52:14.014469 1206 tablet_bootstrap.cc:654] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.014397 1205 rpcz_store.cc:276] 1214 20:52:12.808391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.808454 (+ 63us) service_pool.cc:225] Handling call
1214 20:52:13.993286 (+1184832us) ts_tablet_manager.cc:1151] Deleting tablet d8947ebdc87341ad8b1a01f5dd246172
1214 20:52:13.993300 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.009408 (+ 16108us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.012633 (+ 3225us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.014217 (+ 1584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1184610}
I20251214 20:52:14.014833 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ab4c9e0bdaf74a1791ddad541ece2482" } new_table_name: "default.Table_7c95904dd8b044e4ad472681d5a5a274" modify_external_catalogs: true
I20251214 20:52:14.015972 1314 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" instance_seqno: 1765745526787922) as {username='slave'} at 127.0.104.1:51781; Asking this server to re-register.
I20251214 20:52:14.018635 822 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:14.019233 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 152) took 1794 ms. Trace:
I20251214 20:52:14.019393 1207 rpcz_store.cc:276] 1214 20:52:12.221521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.221571 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:13.761049 (+1539478us) ts_tablet_manager.cc:1151] Deleting tablet f2b66900d6ea42aa849c24acbbe23c40
1214 20:52:13.761058 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:13.889114 (+128056us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.996613 (+107499us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.015875 (+ 19262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1539275}
I20251214 20:52:14.020061 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet f2b66900d6ea42aa849c24acbbe23c40 (table default.Table_934b5fc4d4004b68b7a77ec2cfd47f8c [id=64301732cd3c4699a4a9c49a0adbbfd4]) successfully deleted
I20251214 20:52:14.021150 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 148) took 1800 ms. Trace:
I20251214 20:52:14.021310 1207 rpcz_store.cc:276] 1214 20:52:12.220360 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.221704 (+ 1344us) service_pool.cc:225] Handling call
1214 20:52:14.020421 (+1798717us) ts_tablet_manager.cc:1151] Deleting tablet 6bd6856b800247de896f16eee66f2827
1214 20:52:14.020431 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.021087 (+ 656us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1798207}
I20251214 20:52:14.021787 1202 consensus_queue.cc:260] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.022325 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 153) took 1694 ms. Trace:
I20251214 20:52:14.022517 1202 ts_tablet_manager.cc:1434] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251214 20:52:14.022475 1207 rpcz_store.cc:276] 1214 20:52:12.327361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.327428 (+ 67us) service_pool.cc:225] Handling call
1214 20:52:14.021964 (+1694536us) ts_tablet_manager.cc:1151] Deleting tablet ba50e32f7e724868bb891045a939e362
1214 20:52:14.021975 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.022268 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1694196}
W20251214 20:52:14.023037 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ba50e32f7e724868bb891045a939e362 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba50e32f7e724868bb891045a939e362 already in progress: creating tablet
I20251214 20:52:14.023115 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ecb69e385bf34709b5d954de4b660087 (table default.Table_500b29027c11444893ae5b7a3123cb8b [id=49474ea35419441c9218bc8c896ffd34]) successfully deleted
I20251214 20:52:14.023370 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet d8947ebdc87341ad8b1a01f5dd246172 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]) successfully deleted
I20251214 20:52:14.023862 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 154) took 1619 ms. Trace:
I20251214 20:52:14.023968 1207 rpcz_store.cc:276] 1214 20:52:12.404261 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.404323 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:14.023531 (+1619208us) ts_tablet_manager.cc:1151] Deleting tablet 376e5efede87410eb9f95f36bedebf9d
1214 20:52:14.023541 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.023710 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1618958}
I20251214 20:52:14.024190 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet a063083f54ad49d2a842fa080d44ed85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:14.024382 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 376e5efede87410eb9f95f36bedebf9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 376e5efede87410eb9f95f36bedebf9d already in progress: creating tablet
I20251214 20:52:14.024781 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 155) took 1619 ms. Trace:
I20251214 20:52:14.024892 1207 rpcz_store.cc:276] 1214 20:52:12.404933 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.404998 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:14.024629 (+1619631us) ts_tablet_manager.cc:1151] Deleting tablet d768c45bc02b4d108798efbf3f6aaf7d
1214 20:52:14.024639 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.024732 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1619379}
W20251214 20:52:14.025288 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet d768c45bc02b4d108798efbf3f6aaf7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d768c45bc02b4d108798efbf3f6aaf7d already in progress: creating tablet
I20251214 20:52:14.029307 776 raft_consensus.cc:1275] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Refusing update from remote peer 63848a9fb78a4013a6f1657621272d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 20:52:14.031260 1345 consensus_queue.cc:1048] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:14.041250 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 156) took 1635 ms. Trace:
I20251214 20:52:14.042627 523 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet b0e805cc899347a083d4d549ad68e2f9
I20251214 20:52:14.042970 1499 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251214 20:52:14.043416 1205 tablet_replica.cc:333] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:14.043704 1205 raft_consensus.cc:2243] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.043962 1205 raft_consensus.cc:2272] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.048121 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_58a7d5bd289f408da592c143483b0329"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.048898 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58a7d5bd289f408da592c143483b0329 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.048913 1205 ts_tablet_manager.cc:1916] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.049696 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1c75ca10deb842669e2fea0f16ecc983" } new_table_name: "default.Table_12d421ed905d41eead75b19eb8a793b6" modify_external_catalogs: true
I20251214 20:52:14.056239 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet b0e805cc899347a083d4d549ad68e2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.059666 889 tablet_service.cc:1552] Processing DeleteTablet for tablet b0e805cc899347a083d4d549ad68e2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.060006 1208 ts_tablet_manager.cc:1916] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.061247 1207 rpcz_store.cc:276] 1214 20:52:12.405464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.405516 (+ 52us) service_pool.cc:225] Handling call
1214 20:52:14.041039 (+1635523us) ts_tablet_manager.cc:1151] Deleting tablet 60ae3db968f6412aa0c3952fafe65f88
1214 20:52:14.041051 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.041173 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1634786}
W20251214 20:52:14.062005 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 60ae3db968f6412aa0c3952fafe65f88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60ae3db968f6412aa0c3952fafe65f88 already in progress: creating tablet
I20251214 20:52:14.062893 1207 tablet_replica.cc:333] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:14.063158 1207 raft_consensus.cc:2243] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.063418 1207 raft_consensus.cc:2272] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.064069 1203 tablet_bootstrap.cc:492] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:14.064284 1203 ts_tablet_manager.cc:1403] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.088s user 0.001s sys 0.015s
I20251214 20:52:14.064545 1207 ts_tablet_manager.cc:1916] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.065260 1203 raft_consensus.cc:359] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.065539 1203 raft_consensus.cc:385] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.065673 1203 raft_consensus.cc:740] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.059408 745 tablet_service.cc:1552] Processing DeleteTablet for tablet b0e805cc899347a083d4d549ad68e2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.073086 1206 tablet_bootstrap.cc:492] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:14.073300 1206 ts_tablet_manager.cc:1403] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.003s
I20251214 20:52:14.074138 1206 raft_consensus.cc:359] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.074389 1206 raft_consensus.cc:385] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.074525 1206 raft_consensus.cc:740] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.074846 1206 consensus_queue.cc:260] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.075548 1206 ts_tablet_manager.cc:1434] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:14.075891 1206 tablet_bootstrap.cc:492] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:14.077966 1203 consensus_queue.cc:260] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.078706 1203 ts_tablet_manager.cc:1434] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20251214 20:52:14.081426 1203 tablet_bootstrap.cc:492] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:14.087445 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_08488285c9094a9dacc1e489c50271b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.088169 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08488285c9094a9dacc1e489c50271b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.093168 1205 ts_tablet_manager.cc:1929] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:14.093241 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_0173f37106d54db18ef3975bc6b0d185" } modify_external_catalogs: true
I20251214 20:52:14.093387 1205 log.cc:1199] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ebedf28e17a741169e5490822a2d95df
I20251214 20:52:14.093439 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_0173f37106d54db18ef3975bc6b0d185" } modify_external_catalogs: true
I20251214 20:52:14.093861 1205 ts_tablet_manager.cc:1950] T ebedf28e17a741169e5490822a2d95df P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:14.093909 1465 raft_consensus.cc:493] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.094395 1465 raft_consensus.cc:515] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.094745 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 182) took 1120 ms. Trace:
I20251214 20:52:14.095705 1465 leader_election.cc:290] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:14.096199 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc442726ec0c456290ecf2892d0df957" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:14.098250 1055 raft_consensus.cc:2468] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:14.099078 713 leader_election.cc:304] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:14.099838 915 raft_consensus.cc:1275] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Refusing update from remote peer 63848a9fb78a4013a6f1657621272d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 20:52:14.100590 1202 tablet_bootstrap.cc:492] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:14.100973 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc442726ec0c456290ecf2892d0df957" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:14.101243 911 raft_consensus.cc:2468] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:14.103633 1526 consensus_queue.cc:1048] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:14.108007 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_618d0aa0d8bf4389a544496b923a9db8" } new_table_name: "default.Table_95769572161249d98020c723ac65b78e" modify_external_catalogs: true
I20251214 20:52:14.108196 518 master_service.cc:954] ReplaceTablet: received request to replace tablet a063083f54ad49d2a842fa080d44ed85 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:14.096915 1203 tablet_bootstrap.cc:654] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:14.113160 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb698ff30fcc47d09000cc68d2c5225b consensus state has no leader
I20251214 20:52:14.113580 526 master_service.cc:954] ReplaceTablet: received request to replace tablet c4b22043d81b493fb12116422b1d3fb1 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:14.114645 1205 rpcz_store.cc:276] 1214 20:52:12.973835 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.973908 (+ 73us) service_pool.cc:225] Handling call
1214 20:52:14.043280 (+1069372us) ts_tablet_manager.cc:1151] Deleting tablet ebedf28e17a741169e5490822a2d95df
1214 20:52:14.043297 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.071716 (+ 28419us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.092983 (+ 21267us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.094695 (+ 1712us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039858}
I20251214 20:52:14.115954 1205 tablet_replica.cc:333] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:14.117041 1205 raft_consensus.cc:2243] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.117950 1202 tablet_bootstrap.cc:654] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.118249 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ff05c589ffd8444daa99c956e92c37e6" } modify_external_catalogs: true
I20251214 20:52:14.118523 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ff05c589ffd8444daa99c956e92c37e6" } modify_external_catalogs: true
I20251214 20:52:14.118855 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ebedf28e17a741169e5490822a2d95df (table default.Table_506ad42d4328442997401fa286d1874d [id=08c79b0f6de64d6397fbc34f9aa9fc96]) successfully deleted
I20251214 20:52:14.118923 1510 raft_consensus.cc:493] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.119084 1510 raft_consensus.cc:515] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.119910 1510 leader_election.cc:290] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:14.120685 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0e805cc899347a083d4d549ad68e2f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:14.118575 1205 raft_consensus.cc:2272] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:14.121886 851 leader_election.cc:343] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.124890 1205 ts_tablet_manager.cc:1916] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.131501 1530 tablet.cc:1722] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:14.134853 521 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_cd29381b062249718fbdc39a105f46a7 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 2 got 0
I20251214 20:52:14.135941 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_eb3e055643ea47539ad93faf71ed8cf2" } modify_external_catalogs: true
W20251214 20:52:14.136061 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 956e4b4493f24109ab1251dde0a4cfc0 consensus state has no leader
I20251214 20:52:14.136158 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_eb3e055643ea47539ad93faf71ed8cf2" } modify_external_catalogs: true
I20251214 20:52:14.136593 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_74ac058b83ac4ff3944bce76a31834d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.137349 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74ac058b83ac4ff3944bce76a31834d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.140715 889 tablet_service.cc:1505] Processing CreateTablet for tablet 2accf8a160584c82bb695a52e3f2b05c (DEFAULT_TABLE table=default.Table_d5bf53032a5e4f659230b47347aa809d [id=d31b208e2e284575ad429878129a28f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.141235 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2accf8a160584c82bb695a52e3f2b05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.145875 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_1bba0cb42a0c4aa986d55fdc492ffd46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.146654 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bba0cb42a0c4aa986d55fdc492ffd46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.149595 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 2accf8a160584c82bb695a52e3f2b05c (DEFAULT_TABLE table=default.Table_d5bf53032a5e4f659230b47347aa809d [id=d31b208e2e284575ad429878129a28f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.150146 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2accf8a160584c82bb695a52e3f2b05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.150323 888 tablet_service.cc:1505] Processing CreateTablet for tablet ca57963bcfb54eec9fcbed82757d8387 (DEFAULT_TABLE table=default.Table_04cb0220358347e0b679db3dec97b700 [id=f7d7826ca95442bcac3a73bb3b171956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.150547 1476 raft_consensus.cc:2804] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:14.150758 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca57963bcfb54eec9fcbed82757d8387. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.150722 1476 raft_consensus.cc:493] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:14.157091 1205 ts_tablet_manager.cc:1929] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.157085 880 tablet_service.cc:1505] Processing CreateTablet for tablet ca263a80702f4f64aa7c872c2be08207 (DEFAULT_TABLE table=default.Table_c8d7238b9dc045a7bed9e721f7c022f5 [id=bca3a2d37f8b4fc797a5c52a43c14869]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.157666 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca263a80702f4f64aa7c872c2be08207. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.158877 882 tablet_service.cc:1505] Processing CreateTablet for tablet 4408013cf49e46d0b18d24b48483ea49 (DEFAULT_TABLE table=default.Table_7d6d418a1acb404f908e9ee299cef249 [id=55fc2a6c5e514513904682dfde510d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.159395 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4408013cf49e46d0b18d24b48483ea49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.161907 1208 ts_tablet_manager.cc:1929] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:14.162081 1208 log.cc:1199] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/8140dad1fab142d98b930b53ae5a64c2
I20251214 20:52:14.162469 1208 ts_tablet_manager.cc:1950] T 8140dad1fab142d98b930b53ae5a64c2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.163341 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 172) took 1416 ms. Trace:
I20251214 20:52:14.163444 1208 rpcz_store.cc:276] 1214 20:52:12.747058 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.747159 (+ 101us) service_pool.cc:225] Handling call
1214 20:52:13.996834 (+1249675us) ts_tablet_manager.cc:1151] Deleting tablet 8140dad1fab142d98b930b53ae5a64c2
1214 20:52:13.996845 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.132452 (+135607us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.161755 (+ 29303us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.163275 (+ 1520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1248482}
I20251214 20:52:14.163537 886 tablet_service.cc:1505] Processing CreateTablet for tablet b315744c8cd043c5a8bdceec73a7e28b (DEFAULT_TABLE table=default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.163709 889 tablet_service.cc:1505] Processing CreateTablet for tablet 7dd657f3baff40bb8cb7f4648367f7a1 (DEFAULT_TABLE table=default.Table_8b86c42f7826440982c88a8625559146 [id=c35009dabd4a42598506e52b3d3e493c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.163897 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 8140dad1fab142d98b930b53ae5a64c2 (table default.Table_d6e5176452664864914a6f76f32af5ac [id=3606c9fa9a3d4b55b86dddf293442a3c]) successfully deleted
I20251214 20:52:14.164057 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b315744c8cd043c5a8bdceec73a7e28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.165666 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dd657f3baff40bb8cb7f4648367f7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.167680 878 tablet_service.cc:1505] Processing CreateTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f (DEFAULT_TABLE table=default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.168143 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2e83bca8ed642b6a1b705ef70fde27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.170400 1510 raft_consensus.cc:493] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.170641 1510 raft_consensus.cc:515] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.171473 1510 leader_election.cc:290] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:14.179741 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_69c0719fe94a4c7594516a739f6a40ea" } modify_external_catalogs: true
I20251214 20:52:14.179991 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_69c0719fe94a4c7594516a739f6a40ea" } modify_external_catalogs: true
I20251214 20:52:14.180543 1202 tablet_bootstrap.cc:492] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:14.180747 1202 ts_tablet_manager.cc:1403] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.002s
I20251214 20:52:14.181952 1208 tablet_replica.cc:333] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.157840 1203 tablet_bootstrap.cc:492] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:14.174968 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "956e4b4493f24109ab1251dde0a4cfc0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:14.182272 1055 raft_consensus.cc:2468] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:14.182987 1203 ts_tablet_manager.cc:1403] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.003s
I20251214 20:52:14.183348 1208 raft_consensus.cc:2243] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.184039 1203 raft_consensus.cc:359] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.184369 1203 raft_consensus.cc:385] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.184440 1203 raft_consensus.cc:740] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.184774 1203 consensus_queue.cc:260] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.182889 851 leader_election.cc:304] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:14.185808 1510 raft_consensus.cc:2804] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:14.185925 1510 raft_consensus.cc:493] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:14.186036 1510 raft_consensus.cc:3060] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:14.186914 517 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbbfcef827ee4d548939aaf4783c8109 has no consensus state
I20251214 20:52:14.188805 1510 raft_consensus.cc:515] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.189046 1203 ts_tablet_manager.cc:1434] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251214 20:52:14.189913 1510 leader_election.cc:290] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:14.190651 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "956e4b4493f24109ab1251dde0a4cfc0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:14.181747 1202 raft_consensus.cc:359] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.190966 1055 raft_consensus.cc:3060] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.191053 1202 raft_consensus.cc:385] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.191212 1202 raft_consensus.cc:740] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.191470 1202 consensus_queue.cc:260] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.192130 1202 ts_tablet_manager.cc:1434] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251214 20:52:14.192442 1202 tablet_bootstrap.cc:492] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:14.194051 1202 tablet_bootstrap.cc:654] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.194041 1055 raft_consensus.cc:2468] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:14.195363 851 leader_election.cc:304] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:14.195755 1208 raft_consensus.cc:2272] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.196036 1203 tablet_bootstrap.cc:492] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:14.196962 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 324379e3217442c3addcfe58060b9ad7
I20251214 20:52:14.198014 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d9cef6f07b8743c3aa470da0be52c7c0" } new_table_name: "default.Table_6bac3fed78714367b43db88be8ee1712" modify_external_catalogs: true
I20251214 20:52:14.198693 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 324379e3217442c3addcfe58060b9ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.179463 1205 log.cc:1199] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/119c8cc0d6eb4d7f9a5d7fb3000c347b
I20251214 20:52:14.198904 888 tablet_service.cc:1505] Processing CreateTablet for tablet 6fef1d6add764fd28a8f4995b337ea71 (DEFAULT_TABLE table=default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.199463 1205 ts_tablet_manager.cc:1950] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:14.199705 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fef1d6add764fd28a8f4995b337ea71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.199801 1510 raft_consensus.cc:2804] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:14.200007 1510 raft_consensus.cc:697] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:14.200362 1510 consensus_queue.cc:237] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.203976 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 186) took 1147 ms. Trace:
I20251214 20:52:14.204456 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 24ff58f321124d12b5330dca185d426e
I20251214 20:52:14.204975 520 catalog_manager.cc:6724] ReplaceTablet: tablet 24ff58f321124d12b5330dca185d426e deleted and replaced by tablet 42bdcecba5534294aca1eb419f18e9d2
I20251214 20:52:14.205505 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff58f321124d12b5330dca185d426e with delete_type TABLET_DATA_DELETED (replaced by tablet 42bdcecba5534294aca1eb419f18e9d2) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.205826 1021 tablet_service.cc:1505] Processing CreateTablet for tablet ca57963bcfb54eec9fcbed82757d8387 (DEFAULT_TABLE table=default.Table_04cb0220358347e0b679db3dec97b700 [id=f7d7826ca95442bcac3a73bb3b171956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.206338 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca57963bcfb54eec9fcbed82757d8387. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.206820 1203 tablet_bootstrap.cc:654] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.205981 1205 rpcz_store.cc:276] 1214 20:52:13.056533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.062143 (+ 5610us) service_pool.cc:225] Handling call
1214 20:52:14.115864 (+1053721us) ts_tablet_manager.cc:1151] Deleting tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b
1214 20:52:14.115877 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.153555 (+ 37678us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.156933 (+ 3378us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.203907 (+ 46974us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053206}
I20251214 20:52:14.208402 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 189) took 1101 ms. Trace:
I20251214 20:52:14.208506 1205 rpcz_store.cc:276] 1214 20:52:13.107343 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.107396 (+ 53us) service_pool.cc:225] Handling call
1214 20:52:14.208191 (+1100795us) ts_tablet_manager.cc:1151] Deleting tablet b652a8a80246443491551e76e68d2a8b
1214 20:52:14.208203 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.208340 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1100326}
I20251214 20:52:14.208920 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 190) took 1099 ms. Trace:
I20251214 20:52:14.209017 1205 rpcz_store.cc:276] 1214 20:52:13.109408 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.109457 (+ 49us) service_pool.cc:225] Handling call
1214 20:52:14.208737 (+1099280us) ts_tablet_manager.cc:1151] Deleting tablet 8af99e0bd8774a678575847a9ec2e557
1214 20:52:14.208746 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.208865 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1099069}
I20251214 20:52:14.209281 1205 tablet_replica.cc:333] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:14.209486 1205 raft_consensus.cc:2243] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.209416 878 tablet_service.cc:1505] Processing CreateTablet for tablet f900ebe92efa476cac232131a818d449 (DEFAULT_TABLE table=default.Table_5de120dfaab44b84868db0d1f79e2cee [id=1158f8a25d2b4696a5f5ed2c3a426862]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.209751 1205 raft_consensus.cc:2272] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.209853 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f900ebe92efa476cac232131a818d449. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.210661 1205 ts_tablet_manager.cc:1916] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.213846 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet ebedf28e17a741169e5490822a2d95df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:09 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.213912 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 3fd0fa0dad974c98bf6f52c8cfe83035
I20251214 20:52:14.215021 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd0fa0dad974c98bf6f52c8cfe83035 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.216677 876 tablet_service.cc:1505] Processing CreateTablet for tablet 4aa1a44452074ec28a1bd9c48a5f47cb (DEFAULT_TABLE table=default.Table_dbcc1409fc7e48cb804842a37eebaad2 [id=81c0278b4add47d9947b461866fe9eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.216881 1029 tablet_service.cc:1505] Processing CreateTablet for tablet ca263a80702f4f64aa7c872c2be08207 (DEFAULT_TABLE table=default.Table_c8d7238b9dc045a7bed9e721f7c022f5 [id=bca3a2d37f8b4fc797a5c52a43c14869]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.217101 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa1a44452074ec28a1bd9c48a5f47cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.217330 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca263a80702f4f64aa7c872c2be08207. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.219596 893 tablet_service.cc:1505] Processing CreateTablet for tablet 242a77ae57174de48f4eaf43384ee0ba (DEFAULT_TABLE table=default.Table_9ad469dea994477983bfe62eef37e334 [id=b1be575d16354425ab4f7e8e62847820]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.220120 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 242a77ae57174de48f4eaf43384ee0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.224548 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 4408013cf49e46d0b18d24b48483ea49 (DEFAULT_TABLE table=default.Table_7d6d418a1acb404f908e9ee299cef249 [id=55fc2a6c5e514513904682dfde510d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.225090 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4408013cf49e46d0b18d24b48483ea49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.228367 518 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet a6a47b54175d40118ef23e51a0c46d60
I20251214 20:52:14.198698 1208 ts_tablet_manager.cc:1916] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.236117 1205 ts_tablet_manager.cc:1929] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.237906 1205 log.cc:1199] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/a8e09c13caca4a1c9018387972c9f082
I20251214 20:52:14.238694 1205 ts_tablet_manager.cc:1950] T a8e09c13caca4a1c9018387972c9f082 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:14.240506 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 191) took 1113 ms. Trace:
I20251214 20:52:14.242741 1205 rpcz_store.cc:276] 1214 20:52:13.127210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.127284 (+ 74us) service_pool.cc:225] Handling call
1214 20:52:14.209194 (+1081910us) ts_tablet_manager.cc:1151] Deleting tablet a8e09c13caca4a1c9018387972c9f082
1214 20:52:14.209202 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.225014 (+ 15812us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.235982 (+ 10968us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.240467 (+ 4485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081412}
I20251214 20:52:14.243794 889 tablet_service.cc:1505] Processing CreateTablet for tablet 16816fa6b8c34207bb929147910b93cb (DEFAULT_TABLE table=default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.244251 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16816fa6b8c34207bb929147910b93cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.244975 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 194) took 1049 ms. Trace:
I20251214 20:52:14.245481 1205 rpcz_store.cc:276] 1214 20:52:13.194734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.194793 (+ 59us) service_pool.cc:225] Handling call
1214 20:52:14.244146 (+1049353us) ts_tablet_manager.cc:1151] Deleting tablet 2da66ec83e9140f7a605f0c86abff468
1214 20:52:14.244161 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.244292 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049141}
I20251214 20:52:14.246667 1205 tablet_replica.cc:333] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:14.247429 1205 raft_consensus.cc:2243] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.247674 526 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet c4b22043d81b493fb12116422b1d3fb1
I20251214 20:52:14.248132 526 catalog_manager.cc:6724] ReplaceTablet: tablet c4b22043d81b493fb12116422b1d3fb1 deleted and replaced by tablet ba7952290e2c4746802cffeb14971da0
I20251214 20:52:14.248207 1205 raft_consensus.cc:2272] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.248994 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 7dd657f3baff40bb8cb7f4648367f7a1 (DEFAULT_TABLE table=default.Table_8b86c42f7826440982c88a8625559146 [id=c35009dabd4a42598506e52b3d3e493c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.249485 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dd657f3baff40bb8cb7f4648367f7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.249773 1205 ts_tablet_manager.cc:1916] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.250244 878 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b22043d81b493fb12116422b1d3fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet ba7952290e2c4746802cffeb14971da0) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.250718 893 tablet_service.cc:1505] Processing CreateTablet for tablet 190afbb866854dd7b5393e04443752b7 (DEFAULT_TABLE table=default.Table_501d1ae22d314c82914be72476d1ee0d [id=1f1f307f926f4ce297cf3f9c5423e818]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.251149 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 190afbb866854dd7b5393e04443752b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.253628 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_585da05fa85a4320a84e53b8249d4aaa" } modify_external_catalogs: true
I20251214 20:52:14.253829 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_585da05fa85a4320a84e53b8249d4aaa" } modify_external_catalogs: true
I20251214 20:52:14.254194 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_6838d68b72c042a293fd6c13096e6b78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.254767 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6838d68b72c042a293fd6c13096e6b78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.255558 876 tablet_service.cc:1505] Processing CreateTablet for tablet 012b81223d0f4ae9b742014e329f11a9 (DEFAULT_TABLE table=default.Table_b151b87a38944a7a89d0fe6ff10ae68b [id=7b72926c96404fa5b84cfb86e6ae10a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.255980 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012b81223d0f4ae9b742014e329f11a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.259006 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 324379e3217442c3addcfe58060b9ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.259241 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff58f321124d12b5330dca185d426e with delete_type TABLET_DATA_DELETED (replaced by tablet 42bdcecba5534294aca1eb419f18e9d2) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.263517 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 16816fa6b8c34207bb929147910b93cb (DEFAULT_TABLE table=default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.264082 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16816fa6b8c34207bb929147910b93cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.271694 1205 ts_tablet_manager.cc:1929] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251214 20:52:14.271916 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b652a8a80246443491551e76e68d2a8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b652a8a80246443491551e76e68d2a8b already in progress: creating tablet
W20251214 20:52:14.272190 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8af99e0bd8774a678575847a9ec2e557 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8af99e0bd8774a678575847a9ec2e557 already in progress: creating tablet
I20251214 20:52:14.272424 1205 log.cc:1199] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/f2b66900d6ea42aa849c24acbbe23c40
I20251214 20:52:14.272503 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet a8e09c13caca4a1c9018387972c9f082 (table default.Table_4a73de3f8186423abc9e9123ed3f1acb [id=397be7c374504dc5973e1cbf6b263084]) successfully deleted
I20251214 20:52:14.273325 1205 ts_tablet_manager.cc:1950] T f2b66900d6ea42aa849c24acbbe23c40 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:14.275643 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd0fa0dad974c98bf6f52c8cfe83035 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.276448 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 4aa1a44452074ec28a1bd9c48a5f47cb (DEFAULT_TABLE table=default.Table_dbcc1409fc7e48cb804842a37eebaad2 [id=81c0278b4add47d9947b461866fe9eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.276953 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa1a44452074ec28a1bd9c48a5f47cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.276839 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_cf7fd4bd0b764cc79acc374ea7bb5187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:14.277338 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 195) took 1073 ms. Trace:
W20251214 20:52:14.277460 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf7fd4bd0b764cc79acc374ea7bb5187 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.277443 1205 rpcz_store.cc:276] 1214 20:52:13.202693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.202748 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:14.246585 (+1043837us) ts_tablet_manager.cc:1151] Deleting tablet f2b66900d6ea42aa849c24acbbe23c40
1214 20:52:14.246595 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.260313 (+ 13718us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.270132 (+ 9819us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.276394 (+ 6262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035851}
I20251214 20:52:14.277915 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 198) took 1001 ms. Trace:
I20251214 20:52:14.278004 1205 rpcz_store.cc:276] 1214 20:52:13.276006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.276067 (+ 61us) service_pool.cc:225] Handling call
1214 20:52:14.277745 (+1001678us) ts_tablet_manager.cc:1151] Deleting tablet 3c3962c33a294fa98f8bd485268aee71
1214 20:52:14.277754 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.277853 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1001455}
I20251214 20:52:14.278781 1205 tablet_replica.cc:333] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:14.279006 1205 raft_consensus.cc:2243] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.283061 1205 raft_consensus.cc:2272] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.283763 1205 ts_tablet_manager.cc:1916] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.286442 523 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 4b86f4adaf4540f48a4f42bc45e259e5
W20251214 20:52:14.288486 524 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 144dee1a6d7344f2b79da6e79e67599e consensus state has no leader
I20251214 20:52:14.289778 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.292207 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_e97e8fd3d1924f659ce9738eadcafa9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.293182 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e97e8fd3d1924f659ce9738eadcafa9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.296053 518 catalog_manager.cc:6199] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: default.Table_616b5b16d08b43eb8a04a21a352031fe [id=086ee2aba6724a93b4bb736886fe9615] alter complete (version 1)
I20251214 20:52:14.297008 519 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 4c84cf32c0f94793a2d8b1001ca16b20
I20251214 20:52:14.297019 1513 raft_consensus.cc:493] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.297212 1513 raft_consensus.cc:515] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:14.298094 1513 leader_election.cc:290] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:14.298344 1528 raft_consensus.cc:493] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.298525 1528 raft_consensus.cc:515] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.299348 1528 leader_election.cc:290] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:14.299712 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "569a02142c71427d91868b02695ae12b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:14.300154 851 leader_election.cc:343] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:14.300640 991 leader_election.cc:343] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 118dd44151a34ba3ae58f9fe1d3790d4
I20251214 20:52:14.302753 1208 ts_tablet_manager.cc:1929] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.303041 1208 log.cc:1199] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ecb69e385bf34709b5d954de4b660087
I20251214 20:52:14.303298 1202 tablet_bootstrap.cc:492] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:14.303458 1202 ts_tablet_manager.cc:1403] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.111s user 0.001s sys 0.003s
I20251214 20:52:14.303690 1208 ts_tablet_manager.cc:1950] T ecb69e385bf34709b5d954de4b660087 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.304167 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 6fef1d6add764fd28a8f4995b337ea71 (DEFAULT_TABLE table=default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.304482 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 173) took 1550 ms. Trace:
I20251214 20:52:14.304735 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fef1d6add764fd28a8f4995b337ea71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.305200 1205 ts_tablet_manager.cc:1929] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.311261 1205 log.cc:1199] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/c17126f75b6241c5833269d20774c439
I20251214 20:52:14.312348 1205 ts_tablet_manager.cc:1950] T c17126f75b6241c5833269d20774c439 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:14.314718 1205 tablet_replica.cc:333] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:14.317517 1476 raft_consensus.cc:3060] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.318029 1205 raft_consensus.cc:2243] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.318627 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_c845c10eeec54a7aa19cb8e0cebeb3a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.319272 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c845c10eeec54a7aa19cb8e0cebeb3a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.318226 1205 raft_consensus.cc:2272] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.319926 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_8bb698451aaf41b3a5c13aba022bbb57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:14.320158 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_aec975307a944985bbcc63f028de7eda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.320403 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bb698451aaf41b3a5c13aba022bbb57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:14.320528 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aec975307a944985bbcc63f028de7eda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.321115 1465 consensus_queue.cc:1048] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.217s
I20251214 20:52:14.321432 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_d5bf53032a5e4f659230b47347aa809d" } new_table_name: "default.Table_c3fc0fa4ec254ff889443791279bb8af" modify_external_catalogs: true
I20251214 20:52:14.321703 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_5e8a4451754247c2add9a3245b63b050"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:14.321839 1206 tablet_bootstrap.cc:654] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:14.322130 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e8a4451754247c2add9a3245b63b050 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.326612 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0e805cc899347a083d4d549ad68e2f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:14.328727 1055 raft_consensus.cc:1275] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75 [term 2 FOLLOWER]: Refusing update from remote peer 074399864fb2470fa9e2ee87c1fe2212: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 20:52:14.321870 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "118dd44151a34ba3ae58f9fe1d3790d4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:14.329174 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_c2b912ceb2e64a64a51f04e3cf5d8101 [id=0453c668e17b4c1ba55df9ff8fb1fce7], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 956e4b4493f24109ab1251dde0a4cfc0 consensus state has no leader
I20251214 20:52:14.329427 1460 raft_consensus.cc:493] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.331483 742 tablet_service.cc:1505] Processing CreateTablet for tablet ca57963bcfb54eec9fcbed82757d8387 (DEFAULT_TABLE table=default.Table_04cb0220358347e0b679db3dec97b700 [id=f7d7826ca95442bcac3a73bb3b171956]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.332944 1460 raft_consensus.cc:515] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.335348 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "956e4b4493f24109ab1251dde0a4cfc0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:14.336674 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca57963bcfb54eec9fcbed82757d8387. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:14.337136 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2da66ec83e9140f7a605f0c86abff468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2da66ec83e9140f7a605f0c86abff468 already in progress: creating tablet
W20251214 20:52:14.337754 513 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 64818.8us
I20251214 20:52:14.338267 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet f2b66900d6ea42aa849c24acbbe23c40 (table default.Table_934b5fc4d4004b68b7a77ec2cfd47f8c [id=64301732cd3c4699a4a9c49a0adbbfd4]) successfully deleted
W20251214 20:52:14.338526 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3c3962c33a294fa98f8bd485268aee71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c3962c33a294fa98f8bd485268aee71 already in progress: creating tablet
W20251214 20:52:14.338726 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 52daf1204bf142e7b82852bdec6582fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52daf1204bf142e7b82852bdec6582fb already in progress: creating tablet
I20251214 20:52:14.322062 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "956e4b4493f24109ab1251dde0a4cfc0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:14.340847 745 tablet_service.cc:1505] Processing CreateTablet for tablet 2accf8a160584c82bb695a52e3f2b05c (DEFAULT_TABLE table=default.Table_d5bf53032a5e4f659230b47347aa809d [id=d31b208e2e284575ad429878129a28f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.304337 1202 raft_consensus.cc:359] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:14.322871 886 tablet_service.cc:1505] Processing CreateTablet for tablet ebb26a5ae7224e9ca8a513c7b5013618 (DEFAULT_TABLE table=default.Table_4907bb640d114586b98e6fa0e514cb5a [id=2a6d9d4658f24eb5b39adea101a0891c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:14.330485 848 leader_election.cc:343] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.344511 1202 raft_consensus.cc:385] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.344650 848 leader_election.cc:304] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:14.344717 1202 raft_consensus.cc:740] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.344956 1202 consensus_queue.cc:260] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:14.345237 848 leader_election.cc:343] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.345490 1202 ts_tablet_manager.cc:1434] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
W20251214 20:52:14.345577 848 leader_election.cc:343] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.345783 1202 tablet_bootstrap.cc:492] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:14.346442 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebb26a5ae7224e9ca8a513c7b5013618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.346791 1513 raft_consensus.cc:2749] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:14.342787 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1a95326a5ce4436ca97f62d32966ec22" } new_table_name: "default.Table_55fe32e16f364e58afdd57c80a8966ae" modify_external_catalogs: true
I20251214 20:52:14.348300 1202 tablet_bootstrap.cc:654] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.348551 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_73d965cabee84dd3865221877063447c" } new_table_name: "default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e" modify_external_catalogs: true
W20251214 20:52:14.348647 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 604b7c6fe3b24daca70711a1990af556 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 604b7c6fe3b24daca70711a1990af556 already in progress: creating tablet
W20251214 20:52:14.349205 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 472e4a5990e64d02b1d52dc6e86d7190 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 472e4a5990e64d02b1d52dc6e86d7190 already in progress: creating tablet
I20251214 20:52:14.353654 772 raft_consensus.cc:1240] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [term 2 LEADER]: Rejecting Update request from peer a2e8fd1cb731446d88f8e510c7a52999 for earlier term 1. Current term is 2. Ops: []
I20251214 20:52:14.354506 1533 tablet.cc:1722] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:14.355633 1460 leader_election.cc:290] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:14.356169 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "569a02142c71427d91868b02695ae12b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:14.356734 713 consensus_peers.cc:597] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecb69e385bf34709b5d954de4b660087. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:14.357347 989 leader_election.cc:343] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.357545 989 leader_election.cc:304] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:14.357810 1526 raft_consensus.cc:2749] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:14.358806 756 tablet_service.cc:1505] Processing CreateTablet for tablet 4408013cf49e46d0b18d24b48483ea49 (DEFAULT_TABLE table=default.Table_7d6d418a1acb404f908e9ee299cef249 [id=55fc2a6c5e514513904682dfde510d9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.359243 751 tablet_service.cc:1505] Processing CreateTablet for tablet ca263a80702f4f64aa7c872c2be08207 (DEFAULT_TABLE table=default.Table_c8d7238b9dc045a7bed9e721f7c022f5 [id=bca3a2d37f8b4fc797a5c52a43c14869]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.360045 1205 ts_tablet_manager.cc:1916] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.361909 1476 raft_consensus.cc:515] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.363631 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet c17126f75b6241c5833269d20774c439 (table default.Table_834f1d323b2b4785a85f667e5c14e739 [id=587a162643cf4bc5a3b4c9fc0de83340]) successfully deleted
W20251214 20:52:14.364686 848 leader_election.cc:343] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.364873 848 leader_election.cc:304] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:14.365279 526 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_616b5b16d08b43eb8a04a21a352031fe [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 1 got 0
I20251214 20:52:14.365561 1513 raft_consensus.cc:2749] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:14.365763 1513 consensus_queue.cc:1059] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:14.366047 1513 consensus_queue.cc:1243] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Peer 074399864fb2470fa9e2ee87c1fe2212 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251214 20:52:14.366151 1513 raft_consensus.cc:3055] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Stepping down as leader of term 1
I20251214 20:52:14.366230 1513 raft_consensus.cc:740] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
W20251214 20:52:14.366410 517 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07a8662a634a4befbd2d6251000d2059 consensus state has no leader
I20251214 20:52:14.366519 1513 consensus_queue.cc:260] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:14.366869 1513 raft_consensus.cc:3060] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Advancing to term 2
I20251214 20:52:14.372737 754 tablet_service.cc:1505] Processing CreateTablet for tablet b315744c8cd043c5a8bdceec73a7e28b (DEFAULT_TABLE table=default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.373273 1476 leader_election.cc:290] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:14.373781 1460 consensus_queue.cc:1048] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20251214 20:52:14.375658 749 tablet_service.cc:1505] Processing CreateTablet for tablet 7dd657f3baff40bb8cb7f4648367f7a1 (DEFAULT_TABLE table=default.Table_8b86c42f7826440982c88a8625559146 [id=c35009dabd4a42598506e52b3d3e493c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.376391 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2accf8a160584c82bb695a52e3f2b05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.376547 748 tablet_service.cc:1505] Processing CreateTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f (DEFAULT_TABLE table=default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.344359 1208 rpcz_store.cc:276] 1214 20:52:12.753578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.754038 (+ 460us) service_pool.cc:225] Handling call
1214 20:52:14.181811 (+1427773us) ts_tablet_manager.cc:1151] Deleting tablet ecb69e385bf34709b5d954de4b660087
1214 20:52:14.181824 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.234332 (+ 52508us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.302552 (+ 68220us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.304447 (+ 1895us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1427475}
I20251214 20:52:14.377269 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 174) took 1583 ms. Trace:
I20251214 20:52:14.377449 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2e83bca8ed642b6a1b705ef70fde27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.378010 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff58f321124d12b5330dca185d426e with delete_type TABLET_DATA_DELETED (replaced by tablet 42bdcecba5534294aca1eb419f18e9d2) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.379335 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 324379e3217442c3addcfe58060b9ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.379860 755 tablet_service.cc:1505] Processing CreateTablet for tablet 6fef1d6add764fd28a8f4995b337ea71 (DEFAULT_TABLE table=default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.380131 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ecb69e385bf34709b5d954de4b660087 (table default.Table_500b29027c11444893ae5b7a3123cb8b [id=49474ea35419441c9218bc8c896ffd34]) successfully deleted
I20251214 20:52:14.380223 753 tablet_service.cc:1505] Processing CreateTablet for tablet 16816fa6b8c34207bb929147910b93cb (DEFAULT_TABLE table=default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.380733 750 tablet_service.cc:1505] Processing CreateTablet for tablet 4aa1a44452074ec28a1bd9c48a5f47cb (DEFAULT_TABLE table=default.Table_dbcc1409fc7e48cb804842a37eebaad2 [id=81c0278b4add47d9947b461866fe9eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.381402 746 tablet_service.cc:1505] Processing CreateTablet for tablet f900ebe92efa476cac232131a818d449 (DEFAULT_TABLE table=default.Table_5de120dfaab44b84868db0d1f79e2cee [id=1158f8a25d2b4696a5f5ed2c3a426862]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.381655 747 tablet_service.cc:1505] Processing CreateTablet for tablet 242a77ae57174de48f4eaf43384ee0ba (DEFAULT_TABLE table=default.Table_9ad469dea994477983bfe62eef37e334 [id=b1be575d16354425ab4f7e8e62847820]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.381970 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fd0fa0dad974c98bf6f52c8cfe83035 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.382622 752 tablet_service.cc:1505] Processing CreateTablet for tablet 190afbb866854dd7b5393e04443752b7 (DEFAULT_TABLE table=default.Table_501d1ae22d314c82914be72476d1ee0d [id=1f1f307f926f4ce297cf3f9c5423e818]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.382615 1208 rpcz_store.cc:276] 1214 20:52:12.793836 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.794071 (+ 235us) service_pool.cc:225] Handling call
1214 20:52:14.377091 (+1583020us) ts_tablet_manager.cc:1151] Deleting tablet 12b5f4326aa0413eb303c132ba21decd
1214 20:52:14.377102 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.377218 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1582778}
I20251214 20:52:14.383011 1208 tablet_replica.cc:333] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.383280 1208 raft_consensus.cc:2243] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.383343 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dd657f3baff40bb8cb7f4648367f7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.383486 1208 raft_consensus.cc:2272] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.383606 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fef1d6add764fd28a8f4995b337ea71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.384521 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 190afbb866854dd7b5393e04443752b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.385118 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16816fa6b8c34207bb929147910b93cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.385465 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 242a77ae57174de48f4eaf43384ee0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.386063 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f900ebe92efa476cac232131a818d449. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:14.388301 523 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2accf8a160584c82bb695a52e3f2b05c consensus state has no leader
I20251214 20:52:14.391366 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa1a44452074ec28a1bd9c48a5f47cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.392340 1203 tablet_bootstrap.cc:492] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:14.392526 1203 ts_tablet_manager.cc:1403] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.197s user 0.016s sys 0.022s
I20251214 20:52:14.397603 740 tablet_service.cc:1505] Processing CreateTablet for tablet 012b81223d0f4ae9b742014e329f11a9 (DEFAULT_TABLE table=default.Table_b151b87a38944a7a89d0fe6ff10ae68b [id=7b72926c96404fa5b84cfb86e6ae10a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.398305 1208 ts_tablet_manager.cc:1916] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.398535 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b315744c8cd043c5a8bdceec73a7e28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.398841 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012b81223d0f4ae9b742014e329f11a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.399263 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca263a80702f4f64aa7c872c2be08207. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.399667 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.400161 743 tablet_service.cc:1505] Processing CreateTablet for tablet 4d6e7306254044449f1127fb963ca5f0 (DEFAULT_TABLE table=default.Table_384f071b4b4d411aa4fe7bcd8160a32a [id=fc5b05a963ff4c40a975fb895072265e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.400290 741 tablet_service.cc:1505] Processing CreateTablet for tablet ebb26a5ae7224e9ca8a513c7b5013618 (DEFAULT_TABLE table=default.Table_4907bb640d114586b98e6fa0e514cb5a [id=2a6d9d4658f24eb5b39adea101a0891c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.400534 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d6e7306254044449f1127fb963ca5f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.401113 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebb26a5ae7224e9ca8a513c7b5013618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.401276 1314 ts_manager.cc:194] Registered new tserver with Master: 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:14.401499 739 tablet_service.cc:1505] Processing CreateTablet for tablet d1389716f5ce44f588c21a34c7a7d4c8 (DEFAULT_TABLE table=default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.402100 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1389716f5ce44f588c21a34c7a7d4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.402336 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4408013cf49e46d0b18d24b48483ea49. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.403440 1203 raft_consensus.cc:359] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.403774 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "376e5efede87410eb9f95f36bedebf9d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:14.404071 915 raft_consensus.cc:2468] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
W20251214 20:52:14.404469 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 12b5f4326aa0413eb303c132ba21decd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12b5f4326aa0413eb303c132ba21decd already in progress: creating tablet
I20251214 20:52:14.405083 1203 raft_consensus.cc:385] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.405140 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc442726ec0c456290ecf2892d0df957" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:14.405367 915 raft_consensus.cc:3060] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.405674 1203 raft_consensus.cc:740] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.406623 1203 consensus_queue.cc:260] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.408179 1202 tablet_bootstrap.cc:492] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:14.408370 1202 ts_tablet_manager.cc:1403] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.000s
I20251214 20:52:14.409221 915 raft_consensus.cc:2468] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:14.409194 1202 raft_consensus.cc:359] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.409387 1202 raft_consensus.cc:385] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.409471 1202 raft_consensus.cc:740] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.409734 1202 consensus_queue.cc:260] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.410540 876 tablet_service.cc:1505] Processing CreateTablet for tablet d1389716f5ce44f588c21a34c7a7d4c8 (DEFAULT_TABLE table=default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.413081 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 012b81223d0f4ae9b742014e329f11a9 (DEFAULT_TABLE table=default.Table_b151b87a38944a7a89d0fe6ff10ae68b [id=7b72926c96404fa5b84cfb86e6ae10a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.413552 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012b81223d0f4ae9b742014e329f11a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.413996 744 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b22043d81b493fb12116422b1d3fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet ba7952290e2c4746802cffeb14971da0) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.414893 1203 ts_tablet_manager.cc:1434] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.022s user 0.002s sys 0.002s
I20251214 20:52:14.419431 1203 tablet_bootstrap.cc:492] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:14.419714 712 consensus_peers.cc:597] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecb69e385bf34709b5d954de4b660087. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:14.419961 712 leader_election.cc:304] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:14.420357 712 leader_election.cc:304] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:14.424392 911 raft_consensus.cc:1275] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 2 FOLLOWER]: Refusing update from remote peer 074399864fb2470fa9e2ee87c1fe2212: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251214 20:52:14.424818 1206 tablet_bootstrap.cc:492] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:14.427327 1465 raft_consensus.cc:2804] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:14.438273 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1389716f5ce44f588c21a34c7a7d4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.439589 1020 tablet_service.cc:1505] Processing CreateTablet for tablet ebb26a5ae7224e9ca8a513c7b5013618 (DEFAULT_TABLE table=default.Table_4907bb640d114586b98e6fa0e514cb5a [id=2a6d9d4658f24eb5b39adea101a0891c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.440105 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebb26a5ae7224e9ca8a513c7b5013618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.440236 1202 ts_tablet_manager.cc:1434] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20251214 20:52:14.442871 1202 tablet_bootstrap.cc:492] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:14.443228 1526 raft_consensus.cc:493] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.443400 1526 raft_consensus.cc:515] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.444222 1526 leader_election.cc:290] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:14.445799 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed1d007f1ca24b7291262708ddfca3c3" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:14.446064 911 raft_consensus.cc:2468] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:14.446364 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.446619 991 leader_election.cc:304] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:14.447021 1526 raft_consensus.cc:2804] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:14.447156 1526 raft_consensus.cc:493] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:14.447233 1526 raft_consensus.cc:3060] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.448211 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 242a77ae57174de48f4eaf43384ee0ba (DEFAULT_TABLE table=default.Table_9ad469dea994477983bfe62eef37e334 [id=b1be575d16354425ab4f7e8e62847820]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.448727 1208 ts_tablet_manager.cc:1929] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.448875 1208 log.cc:1199] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/84a0237e4ee34f94821d8647a400575b
I20251214 20:52:14.306681 1030 tablet_service.cc:1505] Processing CreateTablet for tablet f900ebe92efa476cac232131a818d449 (DEFAULT_TABLE table=default.Table_5de120dfaab44b84868db0d1f79e2cee [id=1158f8a25d2b4696a5f5ed2c3a426862]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.451617 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f900ebe92efa476cac232131a818d449. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.451716 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 4d6e7306254044449f1127fb963ca5f0 (DEFAULT_TABLE table=default.Table_384f071b4b4d411aa4fe7bcd8160a32a [id=fc5b05a963ff4c40a975fb895072265e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.452445 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d6e7306254044449f1127fb963ca5f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.308018 1021 tablet_service.cc:1505] Processing CreateTablet for tablet b315744c8cd043c5a8bdceec73a7e28b (DEFAULT_TABLE table=default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.458086 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b315744c8cd043c5a8bdceec73a7e28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.460523 1024 tablet_service.cc:1505] Processing CreateTablet for tablet d1389716f5ce44f588c21a34c7a7d4c8 (DEFAULT_TABLE table=default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.461041 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1389716f5ce44f588c21a34c7a7d4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.463780 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "376e5efede87410eb9f95f36bedebf9d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:14.464033 1055 raft_consensus.cc:2468] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:14.464954 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc442726ec0c456290ecf2892d0df957" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:14.465144 1055 raft_consensus.cc:3060] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.465598 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 242a77ae57174de48f4eaf43384ee0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.467908 1208 ts_tablet_manager.cc:1950] T 84a0237e4ee34f94821d8647a400575b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.468653 1055 raft_consensus.cc:2468] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:14.468904 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 175) took 1674 ms. Trace:
I20251214 20:52:14.469026 1208 rpcz_store.cc:276] 1214 20:52:12.793989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.794351 (+ 362us) service_pool.cc:225] Handling call
1214 20:52:14.382936 (+1588585us) ts_tablet_manager.cc:1151] Deleting tablet 84a0237e4ee34f94821d8647a400575b
1214 20:52:14.382948 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.445585 (+ 62637us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.448606 (+ 3021us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.468838 (+ 20232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1588413}
I20251214 20:52:14.469417 1208 tablet_replica.cc:333] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.469743 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 84a0237e4ee34f94821d8647a400575b (table default.Table_2162a5b765254842bb0b7e4ba7f66136 [id=fa79ab98ae4f405d830ed263d3c0b3c1]) successfully deleted
I20251214 20:52:14.469740 1202 tablet_bootstrap.cc:654] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.472787 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b22043d81b493fb12116422b1d3fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet ba7952290e2c4746802cffeb14971da0) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.449329 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 190afbb866854dd7b5393e04443752b7 (DEFAULT_TABLE table=default.Table_501d1ae22d314c82914be72476d1ee0d [id=1f1f307f926f4ce297cf3f9c5423e818]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.449978 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet dc442726ec0c456290ecf2892d0df957 with delete_type TABLET_DATA_DELETED (replaced by tablet 6bd6856b800247de896f16eee66f2827) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.450215 1526 raft_consensus.cc:515] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.473557 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 190afbb866854dd7b5393e04443752b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.473877 1526 leader_election.cc:290] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:14.474643 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed1d007f1ca24b7291262708ddfca3c3" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:14.474872 911 raft_consensus.cc:3060] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.477772 911 raft_consensus.cc:2468] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:14.306341 1032 tablet_service.cc:1505] Processing CreateTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f (DEFAULT_TABLE table=default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.478255 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2e83bca8ed642b6a1b705ef70fde27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.478384 991 leader_election.cc:304] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:14.478761 1526 raft_consensus.cc:2804] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:14.478912 1526 raft_consensus.cc:697] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:14.479323 1526 consensus_queue.cc:237] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.479557 1208 raft_consensus.cc:2243] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.480197 1208 raft_consensus.cc:2272] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.481401 1208 ts_tablet_manager.cc:1916] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.450373 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 118dd44151a34ba3ae58f9fe1d3790d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.485358 1460 raft_consensus.cc:2804] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:14.485508 1460 raft_consensus.cc:493] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:14.489215 1460 raft_consensus.cc:3060] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.491703 1476 raft_consensus.cc:493] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.493893 1206 ts_tablet_manager.cc:1403] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.349s user 0.001s sys 0.018s
I20251214 20:52:14.497552 1415 consensus_queue.cc:1048] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.070s
I20251214 20:52:14.497779 1415 consensus_queue.cc:1243] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Peer a2e8fd1cb731446d88f8e510c7a52999 log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251214 20:52:14.499332 911 pending_rounds.cc:85] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999: Aborting all ops after (but not including) 0
I20251214 20:52:14.499457 911 pending_rounds.cc:107] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20251214 20:52:14.499665 911 raft_consensus.cc:2889] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251214 20:52:14.500276 1465 raft_consensus.cc:697] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:14.500563 1465 consensus_queue.cc:237] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.502030 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed1d007f1ca24b7291262708ddfca3c3" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:14.515622 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed1d007f1ca24b7291262708ddfca3c3" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:14.522554 1518 mvcc.cc:204] Tried to move back new op lower bound from 7232493708596109312 to 7232493705949290496. Current Snapshot: MvccSnapshot[applied={T|T < 7232493708596109312}]
I20251214 20:52:14.522871 1487 mvcc.cc:204] Tried to move back new op lower bound from 7232493708596109312 to 7232493705949290496. Current Snapshot: MvccSnapshot[applied={T|T < 7232493708596109312}]
I20251214 20:52:14.522897 1530 tablet.cc:1722] T 33a7f355b40f472883e3399209ef6762 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:14.522585 1476 raft_consensus.cc:515] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.530254 772 raft_consensus.cc:2468] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:14.541988 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed1d007f1ca24b7291262708ddfca3c3" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:14.545399 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ed1d007f1ca24b7291262708ddfca3c3" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:14.545743 914 raft_consensus.cc:2393] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:14.547396 1476 leader_election.cc:290] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:14.547976 712 leader_election.cc:304] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:14.552507 1202 tablet_bootstrap.cc:492] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:14.552700 1202 ts_tablet_manager.cc:1403] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.110s user 0.009s sys 0.005s
I20251214 20:52:14.553759 1460 raft_consensus.cc:515] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.555305 1460 leader_election.cc:290] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:14.555521 1460 raft_consensus.cc:3060] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.568980 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "376e5efede87410eb9f95f36bedebf9d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:14.569213 1208 ts_tablet_manager.cc:1929] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.569335 915 raft_consensus.cc:3060] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.569382 1208 log.cc:1199] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/f2b66900d6ea42aa849c24acbbe23c40
I20251214 20:52:14.569815 1208 ts_tablet_manager.cc:1950] T f2b66900d6ea42aa849c24acbbe23c40 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.571831 1202 raft_consensus.cc:359] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.572055 1202 raft_consensus.cc:385] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.572146 1202 raft_consensus.cc:740] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.572409 1202 consensus_queue.cc:260] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.573115 1202 ts_tablet_manager.cc:1434] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20251214 20:52:14.574571 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 176) took 1743 ms. Trace:
I20251214 20:52:14.574734 1208 rpcz_store.cc:276] 1214 20:52:12.831171 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.831233 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:14.469327 (+1638094us) ts_tablet_manager.cc:1151] Deleting tablet f2b66900d6ea42aa849c24acbbe23c40
1214 20:52:14.469337 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.499704 (+ 30367us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.569022 (+ 69318us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.574521 (+ 5499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1637786}
I20251214 20:52:14.575081 1208 tablet_replica.cc:333] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.576335 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet f2b66900d6ea42aa849c24acbbe23c40 (table default.Table_934b5fc4d4004b68b7a77ec2cfd47f8c [id=64301732cd3c4699a4a9c49a0adbbfd4]) successfully deleted
I20251214 20:52:14.576742 1208 raft_consensus.cc:2243] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.576922 1208 raft_consensus.cc:2272] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.577489 1208 ts_tablet_manager.cc:1916] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.577827 915 raft_consensus.cc:2468] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:14.578325 712 leader_election.cc:304] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:14.579041 1476 raft_consensus.cc:2804] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:14.579209 1476 raft_consensus.cc:697] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:14.579849 1476 consensus_queue.cc:237] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.581074 1460 raft_consensus.cc:2749] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:14.581686 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "376e5efede87410eb9f95f36bedebf9d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:14.581923 1055 raft_consensus.cc:3060] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.583535 1202 tablet_bootstrap.cc:492] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:14.585031 1055 raft_consensus.cc:2468] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:14.585359 1202 tablet_bootstrap.cc:654] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:14.586505 1523 log.cc:927] Time spent T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20251214 20:52:14.589025 1549 tablet.cc:1722] T 33a7f355b40f472883e3399209ef6762 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:14.593338 1208 ts_tablet_manager.cc:1929] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.597121 1208 log.cc:1199] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/119c8cc0d6eb4d7f9a5d7fb3000c347b
I20251214 20:52:14.597821 1208 ts_tablet_manager.cc:1950] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.600148 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 185) took 1547 ms. Trace:
I20251214 20:52:14.600726 1208 rpcz_store.cc:276] 1214 20:52:13.053105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.053161 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:14.575011 (+1521850us) ts_tablet_manager.cc:1151] Deleting tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b
1214 20:52:14.575020 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.580774 (+ 5754us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.593186 (+ 12412us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.600072 (+ 6886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1521548}
I20251214 20:52:14.601200 1549 tablet.cc:1722] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:14.601513 1206 raft_consensus.cc:359] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.601743 1206 raft_consensus.cc:385] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.601820 1206 raft_consensus.cc:740] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.602046 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 189) took 1496 ms. Trace:
I20251214 20:52:14.602128 1206 consensus_queue.cc:260] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.602685 1206 ts_tablet_manager.cc:1434] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.109s user 0.002s sys 0.000s
I20251214 20:52:14.603071 1206 tablet_bootstrap.cc:492] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:14.605801 1208 rpcz_store.cc:276] 1214 20:52:13.105505 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.105555 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:14.601844 (+1496289us) ts_tablet_manager.cc:1151] Deleting tablet 8af99e0bd8774a678575847a9ec2e557
1214 20:52:14.601856 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.601992 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1496035}
W20251214 20:52:14.606782 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8af99e0bd8774a678575847a9ec2e557 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8af99e0bd8774a678575847a9ec2e557 already in progress: creating tablet
I20251214 20:52:14.607488 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 188) took 1502 ms. Trace:
I20251214 20:52:14.608134 1208 rpcz_store.cc:276] 1214 20:52:13.105345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.109841 (+ 4496us) service_pool.cc:225] Handling call
1214 20:52:14.607285 (+1497444us) ts_tablet_manager.cc:1151] Deleting tablet b652a8a80246443491551e76e68d2a8b
1214 20:52:14.607300 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.607429 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1497204}
W20251214 20:52:14.608901 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet b652a8a80246443491551e76e68d2a8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b652a8a80246443491551e76e68d2a8b already in progress: creating tablet
I20251214 20:52:14.609231 1208 tablet_replica.cc:333] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.610105 1208 raft_consensus.cc:2243] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.609858 1202 tablet_bootstrap.cc:492] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:14.610337 1202 ts_tablet_manager.cc:1403] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.004s
I20251214 20:52:14.611399 1202 raft_consensus.cc:359] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:14.611732 1202 raft_consensus.cc:385] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.611853 1202 raft_consensus.cc:740] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.612215 1202 consensus_queue.cc:260] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:14.613095 1202 ts_tablet_manager.cc:1434] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:14.613247 1207 ts_tablet_manager.cc:1929] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.613472 1207 log.cc:1199] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/8c3b9ff3c6234507b76afd18f097b6e0
I20251214 20:52:14.613515 1202 tablet_bootstrap.cc:492] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:14.483691 880 tablet_service.cc:1505] Processing CreateTablet for tablet 4d6e7306254044449f1127fb963ca5f0 (DEFAULT_TABLE table=default.Table_384f071b4b4d411aa4fe7bcd8160a32a [id=fc5b05a963ff4c40a975fb895072265e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.614128 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d6e7306254044449f1127fb963ca5f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.615738 1202 tablet_bootstrap.cc:654] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.616014 1208 raft_consensus.cc:2272] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.618543 1208 ts_tablet_manager.cc:1916] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.619544 1207 ts_tablet_manager.cc:1950] T 8c3b9ff3c6234507b76afd18f097b6e0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:14.619984 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 159) took 2082 ms. Trace:
I20251214 20:52:14.620208 1207 rpcz_store.cc:276] 1214 20:52:12.537422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.538958 (+ 1536us) service_pool.cc:225] Handling call
1214 20:52:14.062794 (+1523836us) ts_tablet_manager.cc:1151] Deleting tablet 8c3b9ff3c6234507b76afd18f097b6e0
1214 20:52:14.062806 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.482190 (+419384us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.577915 (+ 95725us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.619942 (+ 42027us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1523538}
I20251214 20:52:14.620893 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 8c3b9ff3c6234507b76afd18f097b6e0 (table default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]) successfully deleted
I20251214 20:52:14.621330 1207 tablet_replica.cc:333] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:14.621595 1207 raft_consensus.cc:2243] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:14.622166 1207 raft_consensus.cc:2272] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.628093 1208 ts_tablet_manager.cc:1929] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251214 20:52:14.630280 713 consensus_peers.cc:597] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c17126f75b6241c5833269d20774c439. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:14.631479 1208 log.cc:1199] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ed1d007f1ca24b7291262708ddfca3c3
I20251214 20:52:14.630810 1202 tablet_bootstrap.cc:492] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:14.631765 1202 ts_tablet_manager.cc:1403] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20251214 20:52:14.632759 1202 raft_consensus.cc:359] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.634848 1202 raft_consensus.cc:385] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.635006 1202 raft_consensus.cc:740] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.635355 1202 consensus_queue.cc:260] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.636065 1202 ts_tablet_manager.cc:1434] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:14.636413 1202 tablet_bootstrap.cc:492] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:14.637557 1208 ts_tablet_manager.cc:1950] T ed1d007f1ca24b7291262708ddfca3c3 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.639669 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 190) took 1508 ms. Trace:
I20251214 20:52:14.640190 1208 rpcz_store.cc:276] 1214 20:52:13.131274 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.134287 (+ 3013us) service_pool.cc:225] Handling call
1214 20:52:14.609145 (+1474858us) ts_tablet_manager.cc:1151] Deleting tablet ed1d007f1ca24b7291262708ddfca3c3
1214 20:52:14.609154 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.621959 (+ 12805us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.627903 (+ 5944us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.639632 (+ 11729us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474644}
I20251214 20:52:14.640811 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ed1d007f1ca24b7291262708ddfca3c3 (table default.Table_e3603e6d524c40dea2a1a88b79d63a8e [id=7febb36ade704d57bdfd4211054277ab]) successfully deleted
I20251214 20:52:14.640098 1202 tablet_bootstrap.cc:654] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.642239 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 193) took 1446 ms. Trace:
I20251214 20:52:14.642419 1208 rpcz_store.cc:276] 1214 20:52:13.195932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.195980 (+ 48us) service_pool.cc:225] Handling call
1214 20:52:14.642020 (+1446040us) ts_tablet_manager.cc:1151] Deleting tablet 2da66ec83e9140f7a605f0c86abff468
1214 20:52:14.642032 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.642187 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1435074}
W20251214 20:52:14.642962 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2da66ec83e9140f7a605f0c86abff468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2da66ec83e9140f7a605f0c86abff468 already in progress: creating tablet
I20251214 20:52:14.642995 1205 ts_tablet_manager.cc:1929] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.643352 1205 log.cc:1199] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/47a67b1b9dc94e498cad91a72e655a75
I20251214 20:52:14.644157 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 196) took 1368 ms. Trace:
I20251214 20:52:14.644317 1208 rpcz_store.cc:276] 1214 20:52:13.276072 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.276148 (+ 76us) service_pool.cc:225] Handling call
1214 20:52:14.643955 (+1367807us) ts_tablet_manager.cc:1151] Deleting tablet 3c3962c33a294fa98f8bd485268aee71
1214 20:52:14.643966 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.644112 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1367482}
W20251214 20:52:14.645020 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3c3962c33a294fa98f8bd485268aee71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c3962c33a294fa98f8bd485268aee71 already in progress: creating tablet
I20251214 20:52:14.645545 1208 tablet_replica.cc:333] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.645817 1208 raft_consensus.cc:2243] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.646086 1208 raft_consensus.cc:2272] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.646214 1513 raft_consensus.cc:493] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.646410 1513 raft_consensus.cc:515] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.647102 1513 leader_election.cc:290] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:14.648156 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023299180d074893847e84dcd9812751" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:14.648380 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023299180d074893847e84dcd9812751" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:14.648996 848 leader_election.cc:343] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:14.649258 851 leader_election.cc:343] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.649392 851 leader_election.cc:304] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:14.649760 1208 ts_tablet_manager.cc:1916] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.649969 1513 raft_consensus.cc:2749] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:14.658263 1514 mvcc.cc:204] Tried to move back new op lower bound from 7232493708596109312 to 7232493705949290496. Current Snapshot: MvccSnapshot[applied={T|T < 7232493708596109312}]
I20251214 20:52:14.658586 1206 tablet_bootstrap.cc:654] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.658864 1533 tablet.cc:1722] T 33a7f355b40f472883e3399209ef6762 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:14.659807 1208 ts_tablet_manager.cc:1929] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.660365 1207 ts_tablet_manager.cc:1916] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:14.661306 525 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35a529702f924a5c9bf53c3c668c6094 consensus state has no leader
I20251214 20:52:14.662900 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_58a7d5bd289f408da592c143483b0329" } modify_external_catalogs: true
I20251214 20:52:14.663182 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_58a7d5bd289f408da592c143483b0329" } modify_external_catalogs: true
I20251214 20:52:14.663290 1208 log.cc:1199] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/dc442726ec0c456290ecf2892d0df957
I20251214 20:52:14.663720 1208 ts_tablet_manager.cc:1950] T dc442726ec0c456290ecf2892d0df957 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.664531 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 197) took 1314 ms. Trace:
I20251214 20:52:14.664655 1208 rpcz_store.cc:276] 1214 20:52:13.350582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.351888 (+ 1306us) service_pool.cc:225] Handling call
1214 20:52:14.645479 (+1293591us) ts_tablet_manager.cc:1151] Deleting tablet dc442726ec0c456290ecf2892d0df957
1214 20:52:14.645488 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.653690 (+ 8202us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.659642 (+ 5952us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.664474 (+ 4832us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293276}
I20251214 20:52:14.665129 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 198) took 1283 ms. Trace:
I20251214 20:52:14.665179 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_f93eb4d354254ec79674c615721834b7" } modify_external_catalogs: true
I20251214 20:52:14.665268 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet dc442726ec0c456290ecf2892d0df957 (table default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]) successfully deleted
I20251214 20:52:14.665403 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_f93eb4d354254ec79674c615721834b7" } modify_external_catalogs: true
I20251214 20:52:14.666785 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1d198a3975c24b84aaeb82c266c746b8" } modify_external_catalogs: true
I20251214 20:52:14.667030 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1d198a3975c24b84aaeb82c266c746b8" } modify_external_catalogs: true
I20251214 20:52:14.667301 1208 rpcz_store.cc:276] 1214 20:52:13.381447 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.381509 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:14.664949 (+1283440us) ts_tablet_manager.cc:1151] Deleting tablet 52daf1204bf142e7b82852bdec6582fb
1214 20:52:14.664958 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.665080 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1283115}
I20251214 20:52:14.667384 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ec65ee1eda49435895dab50f78b4ee15" } modify_external_catalogs: true
I20251214 20:52:14.667531 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ec65ee1eda49435895dab50f78b4ee15" } modify_external_catalogs: true
W20251214 20:52:14.667981 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 52daf1204bf142e7b82852bdec6582fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52daf1204bf142e7b82852bdec6582fb already in progress: creating tablet
I20251214 20:52:14.668102 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_63c9838172164e519ff318d7f570da85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:14.668187 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_727db60cbb7c435d986f772214b16166" } new_table_name: "default.Table_03ec8d3fa1ea4120b178f12195e8c38a" modify_external_catalogs: true
I20251214 20:52:14.668663 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 199) took 1254 ms. Trace:
W20251214 20:52:14.668699 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63c9838172164e519ff318d7f570da85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.668763 1208 rpcz_store.cc:276] 1214 20:52:13.414314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.414372 (+ 58us) service_pool.cc:225] Handling call
1214 20:52:14.668495 (+1254123us) ts_tablet_manager.cc:1151] Deleting tablet 604b7c6fe3b24daca70711a1990af556
1214 20:52:14.668506 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.668613 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1253863}
W20251214 20:52:14.669198 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 604b7c6fe3b24daca70711a1990af556 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 604b7c6fe3b24daca70711a1990af556 already in progress: creating tablet
I20251214 20:52:14.670086 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 201) took 1215 ms. Trace:
I20251214 20:52:14.666780 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_6f66a268b27e4f9b8d09597224149553"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:14.670449 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_04cb0220358347e0b679db3dec97b700" } new_table_name: "default.Table_3e8e22029bad4918b223e4c2fbf75c03" modify_external_catalogs: true
I20251214 20:52:14.670516 1208 rpcz_store.cc:276] 1214 20:52:13.454490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.454546 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:14.669934 (+1215388us) ts_tablet_manager.cc:1151] Deleting tablet 472e4a5990e64d02b1d52dc6e86d7190
1214 20:52:14.669944 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.670036 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1214915}
W20251214 20:52:14.670706 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f66a268b27e4f9b8d09597224149553 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:14.670929 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 472e4a5990e64d02b1d52dc6e86d7190 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 472e4a5990e64d02b1d52dc6e86d7190 already in progress: creating tablet
I20251214 20:52:14.671481 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 205) took 1141 ms. Trace:
I20251214 20:52:14.671895 1208 rpcz_store.cc:276] 1214 20:52:13.529581 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.529646 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:14.671339 (+1141693us) ts_tablet_manager.cc:1151] Deleting tablet a0279992d1f2419ab1caae116a0391b2
1214 20:52:14.671348 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.671432 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141429}
I20251214 20:52:14.672279 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 376e5efede87410eb9f95f36bedebf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:14.672485 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a0279992d1f2419ab1caae116a0391b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0279992d1f2419ab1caae116a0391b2 already in progress: creating tablet
I20251214 20:52:14.673630 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_a60363e46077464b968f9187a8c7a8fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.674345 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a60363e46077464b968f9187a8c7a8fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.674604 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 207) took 1098 ms. Trace:
I20251214 20:52:14.674921 1208 rpcz_store.cc:276] 1214 20:52:13.576464 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.576532 (+ 68us) service_pool.cc:225] Handling call
1214 20:52:14.672776 (+1096244us) ts_tablet_manager.cc:1151] Deleting tablet cd88d1e95ee04ef596863e1ae3687de0
1214 20:52:14.672785 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.674547 (+ 1762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1095975}
W20251214 20:52:14.675452 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet cd88d1e95ee04ef596863e1ae3687de0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd88d1e95ee04ef596863e1ae3687de0 already in progress: creating tablet
I20251214 20:52:14.675537 1208 tablet_replica.cc:333] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.675740 1208 raft_consensus.cc:2243] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.676398 1208 raft_consensus.cc:2272] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.677561 1208 ts_tablet_manager.cc:1916] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.687913 889 tablet_service.cc:1505] Processing CreateTablet for tablet 2d164747cb294bd4b126b4420231867d (DEFAULT_TABLE table=default.Table_74ac058b83ac4ff3944bce76a31834d6 [id=57197e3100144ed8888c7c06dfcd00a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.687914 882 tablet_service.cc:1505] Processing CreateTablet for tablet 42bdcecba5534294aca1eb419f18e9d2 (DEFAULT_TABLE table=default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.688426 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d164747cb294bd4b126b4420231867d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.688973 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 98105bceda9b44bf893f82a4234a3c81 (DEFAULT_TABLE table=default.Table_e532e5fa17f2476a90df03eb6449a067 [id=ece0780edb0f41b787b46b290c4c0515]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.689047 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 2d164747cb294bd4b126b4420231867d (DEFAULT_TABLE table=default.Table_74ac058b83ac4ff3944bce76a31834d6 [id=57197e3100144ed8888c7c06dfcd00a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.689567 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98105bceda9b44bf893f82a4234a3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.689810 518 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]). Expected version 1 got 0
I20251214 20:52:14.690130 518 catalog_manager.cc:5654] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:14.690522 518 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_c2b912ceb2e64a64a51f04e3cf5d8101 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 3 got 0
I20251214 20:52:14.691848 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d164747cb294bd4b126b4420231867d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.692008 886 tablet_service.cc:1505] Processing CreateTablet for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf (DEFAULT_TABLE table=default.Table_08488285c9094a9dacc1e489c50271b3 [id=d3b4eef101ec491485c1889f08de9992]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.692420 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.695509 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bdcecba5534294aca1eb419f18e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.695468 893 tablet_service.cc:1505] Processing CreateTablet for tablet 98105bceda9b44bf893f82a4234a3c81 (DEFAULT_TABLE table=default.Table_e532e5fa17f2476a90df03eb6449a067 [id=ece0780edb0f41b787b46b290c4c0515]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.695915 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98105bceda9b44bf893f82a4234a3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.697475 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 00cf456eda464f5ba22d6893f73b5cba (DEFAULT_TABLE table=default.Table_6838d68b72c042a293fd6c13096e6b78 [id=065328c700574078a1380f42b9848e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.697899 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00cf456eda464f5ba22d6893f73b5cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.698975 1026 tablet_service.cc:1505] Processing CreateTablet for tablet d9f198ef4dfa4f5791144b9a24fe693f (DEFAULT_TABLE table=default.Table_3736aabecd40417f9cb502ac612bacee [id=fc592ea105f242a299337c9cbe438778]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.699460 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f198ef4dfa4f5791144b9a24fe693f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.699419 888 tablet_service.cc:1505] Processing CreateTablet for tablet cbbfcef827ee4d548939aaf4783c8109 (DEFAULT_TABLE table=default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.699748 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbfcef827ee4d548939aaf4783c8109. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.703475 878 tablet_service.cc:1505] Processing CreateTablet for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 (DEFAULT_TABLE table=default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.703887 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.705070 1018 tablet_service.cc:1505] Processing CreateTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 (DEFAULT_TABLE table=default.Table_1bba0cb42a0c4aa986d55fdc492ffd46 [id=ce55ad7e80fe40e99895c52abb55e986]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.705464 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf8643c7e4094f529cd55656c7b3edb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.707593 1020 tablet_service.cc:1505] Processing CreateTablet for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf (DEFAULT_TABLE table=default.Table_08488285c9094a9dacc1e489c50271b3 [id=d3b4eef101ec491485c1889f08de9992]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.708125 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.709477 892 tablet_service.cc:1505] Processing CreateTablet for tablet 00cf456eda464f5ba22d6893f73b5cba (DEFAULT_TABLE table=default.Table_6838d68b72c042a293fd6c13096e6b78 [id=065328c700574078a1380f42b9848e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.710057 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00cf456eda464f5ba22d6893f73b5cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.710391 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_b0d8d299bd0b4950948be859b03a3731"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.710930 523 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a7fc15882e2474bb6885070be13b1fa consensus state has no leader
W20251214 20:52:14.711215 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0d8d299bd0b4950948be859b03a3731 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.711355 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_e97e8fd3d1924f659ce9738eadcafa9e" } modify_external_catalogs: true
I20251214 20:52:14.711519 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_e97e8fd3d1924f659ce9738eadcafa9e" } modify_external_catalogs: true
I20251214 20:52:14.712965 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3369f89218b140e7a52da8221726f4e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:14.713265 883 tablet_service.cc:1505] Processing CreateTablet for tablet d9f198ef4dfa4f5791144b9a24fe693f (DEFAULT_TABLE table=default.Table_3736aabecd40417f9cb502ac612bacee [id=fc592ea105f242a299337c9cbe438778]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:14.713578 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3369f89218b140e7a52da8221726f4e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.713716 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f198ef4dfa4f5791144b9a24fe693f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.715022 1513 raft_consensus.cc:493] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.715289 1513 raft_consensus.cc:515] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:14.716131 1513 leader_election.cc:290] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:14.717067 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e2ee5cc95304fc1a80ed4d31838ac38" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:14.717790 884 tablet_service.cc:1505] Processing CreateTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 (DEFAULT_TABLE table=default.Table_1bba0cb42a0c4aa986d55fdc492ffd46 [id=ce55ad7e80fe40e99895c52abb55e986]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.718573 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf8643c7e4094f529cd55656c7b3edb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.720850 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 42bdcecba5534294aca1eb419f18e9d2 (DEFAULT_TABLE table=default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:14.721156 525 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca57963bcfb54eec9fcbed82757d8387 consensus state has no leader
I20251214 20:52:14.721325 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bdcecba5534294aca1eb419f18e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.721648 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_afb01d0257eb4d95bff481e5c900da2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.722324 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afb01d0257eb4d95bff481e5c900da2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.724099 1021 tablet_service.cc:1505] Processing CreateTablet for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 (DEFAULT_TABLE table=default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.724576 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.724591 891 tablet_service.cc:1505] Processing CreateTablet for tablet 738668adf2874d6894db6fa875f6990c (DEFAULT_TABLE table=default.Table_b92e50faaffb45c0a3e1c7a94166f985 [id=406157031d694ada8a9fcba4e4e23e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.725034 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 738668adf2874d6894db6fa875f6990c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:14.727898 851 leader_election.cc:343] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.728955 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 738668adf2874d6894db6fa875f6990c (DEFAULT_TABLE table=default.Table_b92e50faaffb45c0a3e1c7a94166f985 [id=406157031d694ada8a9fcba4e4e23e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.729485 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 738668adf2874d6894db6fa875f6990c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.737213 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 8faa7fd7315f485c82aee38bcf4cd8fc
I20251214 20:52:14.738292 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_025651c33fe34ab1a55af7368f32fa04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.738986 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_025651c33fe34ab1a55af7368f32fa04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.739856 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_c2b912ceb2e64a64a51f04e3cf5d8101" } new_table_name: "default.Table_abc73c8e0e7e458abdfb7be16dd324e3" modify_external_catalogs: true
I20251214 20:52:14.740558 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet a10663bd306d40e7a6375d06d69be22d
I20251214 20:52:14.741371 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3922bb23d5a34e7ab2e51a997856bffb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.742007 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3922bb23d5a34e7ab2e51a997856bffb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.742986 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 9c0f49e5d78240a296e47eccb3218e4f
I20251214 20:52:14.743920 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_c8d7238b9dc045a7bed9e721f7c022f5" } new_table_name: "default.Table_642da83f7ed0444688afdf91b7a9e7e8" modify_external_catalogs: true
I20251214 20:52:14.744843 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.746095 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_b3f5ab40d9334de1a194aab3347387d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.746745 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3f5ab40d9334de1a194aab3347387d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.747155 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.747298 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.748431 523 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet be8ceb6205c942929a40dad26ce9b625
I20251214 20:52:14.748692 523 master_service.cc:954] ReplaceTablet: received request to replace tablet b1df0dad2fb040a5a12e0401e00c87db from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:14.749207 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 63ef97aeeaa94f4da683fbd133b05bd1 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:14.749305 520 catalog_manager.cc:6199] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: default.Table_12d421ed905d41eead75b19eb8a793b6 [id=5272550343144d4baaba07e97ebaa187] alter complete (version 2)
I20251214 20:52:14.749761 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 5da9deeedce0421c9aeaebd71ba11b1b from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:14.754961 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.757688 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet a10663bd306d40e7a6375d06d69be22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.757961 893 tablet_service.cc:1552] Processing DeleteTablet for tablet a10663bd306d40e7a6375d06d69be22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.758708 1020 tablet_service.cc:1505] Processing CreateTablet for tablet ec85857f047e44d78835305ddcb70768 (DEFAULT_TABLE table=default.Table_58a7d5bd289f408da592c143483b0329 [id=bb060058712440c7b1e5c40332a0263d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.759248 893 tablet_service.cc:1505] Processing CreateTablet for tablet ec85857f047e44d78835305ddcb70768 (DEFAULT_TABLE table=default.Table_58a7d5bd289f408da592c143483b0329 [id=bb060058712440c7b1e5c40332a0263d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.759404 526 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet ec85857f047e44d78835305ddcb70768
I20251214 20:52:14.760413 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec85857f047e44d78835305ddcb70768. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.760386 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 35a529702f924a5c9bf53c3c668c6094 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:14.760958 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec85857f047e44d78835305ddcb70768. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.763270 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet ec85857f047e44d78835305ddcb70768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.767343 892 tablet_service.cc:1552] Processing DeleteTablet for tablet ec85857f047e44d78835305ddcb70768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.772461 525 master_service.cc:954] ReplaceTablet: received request to replace tablet ee7d709316b24b8eb223a838710c2979 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:14.773757 1017 tablet_service.cc:1505] Processing CreateTablet for tablet cbbfcef827ee4d548939aaf4783c8109 (DEFAULT_TABLE table=default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.774343 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbfcef827ee4d548939aaf4783c8109. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.778738 523 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 63ef97aeeaa94f4da683fbd133b05bd1
W20251214 20:52:14.779428 524 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca263a80702f4f64aa7c872c2be08207 consensus state has no leader
I20251214 20:52:14.779659 523 catalog_manager.cc:6724] ReplaceTablet: tablet 63ef97aeeaa94f4da683fbd133b05bd1 deleted and replaced by tablet 9b1a937150ce4ecfa210c8465d682496
I20251214 20:52:14.780164 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 5da9deeedce0421c9aeaebd71ba11b1b
I20251214 20:52:14.780649 520 catalog_manager.cc:6724] ReplaceTablet: tablet 5da9deeedce0421c9aeaebd71ba11b1b deleted and replaced by tablet 1dca369dc5f54a4ea1783e334c5670d6
I20251214 20:52:14.781615 1202 tablet_bootstrap.cc:492] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:14.781821 1202 ts_tablet_manager.cc:1403] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.145s user 0.006s sys 0.013s
I20251214 20:52:14.782788 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 63ef97aeeaa94f4da683fbd133b05bd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b1a937150ce4ecfa210c8465d682496) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.783221 1202 raft_consensus.cc:359] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.783442 1202 raft_consensus.cc:385] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:14.783516 1202 raft_consensus.cc:740] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:14.783767 1202 consensus_queue.cc:260] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.785211 520 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet f0d661a1bb034053a4634ff537eafc7d (table default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90]). Expected version 1 got 0
I20251214 20:52:14.785564 520 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 144dee1a6d7344f2b79da6e79e67599e (table default.Table_6bac3fed78714367b43db88be8ee1712 [id=8cd62da3d57c4be4b5f10ace49d2e148]). Expected version 3 got 0
I20251214 20:52:14.785760 1202 ts_tablet_manager.cc:1434] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:14.786096 1202 tablet_bootstrap.cc:492] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:14.787055 520 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet f0d661a1bb034053a4634ff537eafc7d (table default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f0d661a1bb034053a4634ff537eafc7d consensus state has no leader
I20251214 20:52:14.787393 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 1dca369dc5f54a4ea1783e334c5670d6) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.787756 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cfd9b2f9756433e9e52643417c4cd45) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.788103 1202 tablet_bootstrap.cc:654] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.793612 1208 ts_tablet_manager.cc:1929] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.794479 1208 log.cc:1199] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/376e5efede87410eb9f95f36bedebf9d
I20251214 20:52:14.795023 1208 ts_tablet_manager.cc:1950] T 376e5efede87410eb9f95f36bedebf9d P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.795974 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 210) took 1160 ms. Trace:
I20251214 20:52:14.796197 1208 rpcz_store.cc:276] 1214 20:52:13.635200 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.635281 (+ 81us) service_pool.cc:225] Handling call
1214 20:52:14.675457 (+1040176us) ts_tablet_manager.cc:1151] Deleting tablet 376e5efede87410eb9f95f36bedebf9d
1214 20:52:14.675467 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.681285 (+ 5818us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.793450 (+112165us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.795917 (+ 2467us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039859}
I20251214 20:52:14.796831 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 376e5efede87410eb9f95f36bedebf9d (table default.Table_2dab958dcddb48e28bfc246163f539ae [id=0b01345b8b0b44c0b59bced8e8b89a75]) successfully deleted
I20251214 20:52:14.797268 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 810ba69dda7f4631b1432bd99029609b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.797482 878 tablet_service.cc:1505] Processing CreateTablet for tablet ba7952290e2c4746802cffeb14971da0 (DEFAULT_TABLE table=default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.799253 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 1dca369dc5f54a4ea1783e334c5670d6) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.799994 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba7952290e2c4746802cffeb14971da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.800257 893 tablet_service.cc:1505] Processing CreateTablet for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (DEFAULT_TABLE table=default.Table_f487575ab59448afa92cbbb86139dd53 [id=52a7552cd4b44ca68f0ad5f1538a99a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.800704 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658e4fe0d188483b9ceb9eb3b3f20835. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.800189 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (DEFAULT_TABLE table=default.Table_f487575ab59448afa92cbbb86139dd53 [id=52a7552cd4b44ca68f0ad5f1538a99a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.796397 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b4f582d8165439d86d9228c146dc41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.801796 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658e4fe0d188483b9ceb9eb3b3f20835. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.804971 1020 tablet_service.cc:1505] Processing CreateTablet for tablet ba7952290e2c4746802cffeb14971da0 (DEFAULT_TABLE table=default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.806648 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba7952290e2c4746802cffeb14971da0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:14.809644 848 consensus_peers.cc:597] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:14.810673 520 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]). Expected version 3 got 0
I20251214 20:52:14.814101 526 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 35a529702f924a5c9bf53c3c668c6094
I20251214 20:52:14.814574 526 catalog_manager.cc:6724] ReplaceTablet: tablet 35a529702f924a5c9bf53c3c668c6094 deleted and replaced by tablet 719d157ac8f74eb7911df05ab5e4a3d5
I20251214 20:52:14.815311 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 35a529702f924a5c9bf53c3c668c6094 with delete_type TABLET_DATA_DELETED (replaced by tablet 719d157ac8f74eb7911df05ab5e4a3d5) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:14.815631 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 214) took 1120 ms. Trace:
I20251214 20:52:14.815878 1208 rpcz_store.cc:276] 1214 20:52:13.695554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.695612 (+ 58us) service_pool.cc:225] Handling call
1214 20:52:14.815423 (+1119811us) ts_tablet_manager.cc:1151] Deleting tablet a34eb8fa3e34427495fa5985b7ba81bf
1214 20:52:14.815437 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.815584 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119574}
W20251214 20:52:14.816450 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a34eb8fa3e34427495fa5985b7ba81bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a34eb8fa3e34427495fa5985b7ba81bf already in progress: creating tablet
I20251214 20:52:14.816919 1208 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43856 (request call id 219) took 1001 ms. Trace:
I20251214 20:52:14.820019 1208 rpcz_store.cc:276] 1214 20:52:13.815777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.815879 (+ 102us) service_pool.cc:225] Handling call
1214 20:52:14.816776 (+1000897us) ts_tablet_manager.cc:1151] Deleting tablet d6c0f445cd7e44769d79ad5566fd8d97
1214 20:52:14.816787 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.816880 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1000622}
W20251214 20:52:14.822552 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet d6c0f445cd7e44769d79ad5566fd8d97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6c0f445cd7e44769d79ad5566fd8d97 already in progress: creating tablet
W20251214 20:52:14.823709 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 40a9a6192c744282ab5672cbe15fd299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40a9a6192c744282ab5672cbe15fd299 already in progress: creating tablet
W20251214 20:52:14.824378 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ee7d709316b24b8eb223a838710c2979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7d709316b24b8eb223a838710c2979 already in progress: creating tablet
W20251214 20:52:14.825198 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a063083f54ad49d2a842fa080d44ed85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a063083f54ad49d2a842fa080d44ed85 already in progress: creating tablet
I20251214 20:52:14.825768 1208 tablet_replica.cc:333] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.828173 1208 raft_consensus.cc:2243] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.828460 1208 raft_consensus.cc:2272] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.829672 1208 ts_tablet_manager.cc:1916] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.836803 1208 ts_tablet_manager.cc:1929] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.837083 1208 log.cc:1199] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/b0e805cc899347a083d4d549ad68e2f9
I20251214 20:52:14.837544 1208 ts_tablet_manager.cc:1950] T b0e805cc899347a083d4d549ad68e2f9 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.838889 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet b0e805cc899347a083d4d549ad68e2f9 (table default.Table_2695eb1d7e3a40c08c5a19eb6afce20f [id=ae6d8ad2d21d4208a1c19435e108f95b]) successfully deleted
I20251214 20:52:14.839799 1055 raft_consensus.cc:1275] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Refusing update from remote peer a2e8fd1cb731446d88f8e510c7a52999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 20:52:14.840668 1510 consensus_queue.cc:1048] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:14.842391 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 324379e3217442c3addcfe58060b9ad7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 324379e3217442c3addcfe58060b9ad7 already in progress: creating tablet
I20251214 20:52:14.843436 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 63ef97aeeaa94f4da683fbd133b05bd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b1a937150ce4ecfa210c8465d682496) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:14.843986 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 24ff58f321124d12b5330dca185d426e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ff58f321124d12b5330dca185d426e already in progress: creating tablet
I20251214 20:52:14.845314 1208 tablet_replica.cc:333] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.845602 1208 raft_consensus.cc:2243] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.845847 1208 raft_consensus.cc:2272] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.846504 1208 ts_tablet_manager.cc:1916] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.849870 1510 raft_consensus.cc:493] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.850049 1510 raft_consensus.cc:515] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.850672 1510 leader_election.cc:290] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:14.852897 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "810ba69dda7f4631b1432bd99029609b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:14.853792 851 leader_election.cc:343] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.857550 1208 ts_tablet_manager.cc:1929] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.857733 1208 log.cc:1199] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/3fd0fa0dad974c98bf6f52c8cfe83035
I20251214 20:52:14.858335 1208 ts_tablet_manager.cc:1950] T 3fd0fa0dad974c98bf6f52c8cfe83035 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.858537 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 35a529702f924a5c9bf53c3c668c6094 with delete_type TABLET_DATA_DELETED (replaced by tablet 719d157ac8f74eb7911df05ab5e4a3d5) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:14.859768 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 3fd0fa0dad974c98bf6f52c8cfe83035 (table default.Table_0173f37106d54db18ef3975bc6b0d185 [id=5c14810dd5364dcaa218b80c3756eadc]) successfully deleted
W20251214 20:52:14.863191 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c4b22043d81b493fb12116422b1d3fb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4b22043d81b493fb12116422b1d3fb1 already in progress: creating tablet
W20251214 20:52:14.863801 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 4b86f4adaf4540f48a4f42bc45e259e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b86f4adaf4540f48a4f42bc45e259e5 already in progress: creating tablet
W20251214 20:52:14.864518 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 9c0f49e5d78240a296e47eccb3218e4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c0f49e5d78240a296e47eccb3218e4f already in progress: creating tablet
W20251214 20:52:14.865142 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8faa7fd7315f485c82aee38bcf4cd8fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8faa7fd7315f485c82aee38bcf4cd8fc already in progress: creating tablet
W20251214 20:52:14.865767 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a10663bd306d40e7a6375d06d69be22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a10663bd306d40e7a6375d06d69be22d already in progress: creating tablet
W20251214 20:52:14.866348 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ec85857f047e44d78835305ddcb70768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec85857f047e44d78835305ddcb70768 already in progress: creating tablet
W20251214 20:52:14.867626 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 63ef97aeeaa94f4da683fbd133b05bd1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63ef97aeeaa94f4da683fbd133b05bd1 already in progress: creating tablet
W20251214 20:52:14.868162 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 5da9deeedce0421c9aeaebd71ba11b1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5da9deeedce0421c9aeaebd71ba11b1b already in progress: creating tablet
I20251214 20:52:14.868470 1208 tablet_replica.cc:333] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.870172 1208 raft_consensus.cc:2243] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.870541 1208 raft_consensus.cc:2272] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.871263 1208 ts_tablet_manager.cc:1916] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:14.683215 745 tablet_service.cc:1552] Processing DeleteTablet for tablet ed1d007f1ca24b7291262708ddfca3c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.683964 739 tablet_service.cc:1505] Processing CreateTablet for tablet 2d164747cb294bd4b126b4420231867d (DEFAULT_TABLE table=default.Table_74ac058b83ac4ff3944bce76a31834d6 [id=57197e3100144ed8888c7c06dfcd00a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.876667 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d164747cb294bd4b126b4420231867d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.686676 754 tablet_service.cc:1552] Processing DeleteTablet for tablet dc442726ec0c456290ecf2892d0df957 with delete_type TABLET_DATA_DELETED (replaced by tablet 6bd6856b800247de896f16eee66f2827) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.699520 748 tablet_service.cc:1505] Processing CreateTablet for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf (DEFAULT_TABLE table=default.Table_08488285c9094a9dacc1e489c50271b3 [id=d3b4eef101ec491485c1889f08de9992]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.718601 740 tablet_service.cc:1505] Processing CreateTablet for tablet 00cf456eda464f5ba22d6893f73b5cba (DEFAULT_TABLE table=default.Table_6838d68b72c042a293fd6c13096e6b78 [id=065328c700574078a1380f42b9848e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.720199 750 tablet_service.cc:1505] Processing CreateTablet for tablet d9f198ef4dfa4f5791144b9a24fe693f (DEFAULT_TABLE table=default.Table_3736aabecd40417f9cb502ac612bacee [id=fc592ea105f242a299337c9cbe438778]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.720670 741 tablet_service.cc:1505] Processing CreateTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 (DEFAULT_TABLE table=default.Table_1bba0cb42a0c4aa986d55fdc492ffd46 [id=ce55ad7e80fe40e99895c52abb55e986]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.724442 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e2ee5cc95304fc1a80ed4d31838ac38" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:14.884689 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.886485 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1ec14d4074ea4e058dcb928e5eddf5b2" } new_table_name: "default.Table_7b427a00182f4e059423bf0e18859b90" modify_external_catalogs: true
I20251214 20:52:14.887827 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf8643c7e4094f529cd55656c7b3edb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.892340 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_44bb60b4a8ec45329622c851c90b5aa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.892938 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44bb60b4a8ec45329622c851c90b5aa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.892995 1203 tablet_bootstrap.cc:654] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:14.894340 1205 ts_tablet_manager.cc:1950] T 47a67b1b9dc94e498cad91a72e655a75 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:14.894665 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00cf456eda464f5ba22d6893f73b5cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.895450 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 207) took 1413 ms. Trace:
I20251214 20:52:14.895581 1205 rpcz_store.cc:276] 1214 20:52:13.481783 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.481844 (+ 61us) service_pool.cc:225] Handling call
1214 20:52:14.314639 (+832795us) ts_tablet_manager.cc:1151] Deleting tablet 47a67b1b9dc94e498cad91a72e655a75
1214 20:52:14.314649 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.398022 (+ 83373us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.634215 (+236193us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.895378 (+261163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":832523}
I20251214 20:52:14.896272 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 209) took 1366 ms. Trace:
I20251214 20:52:14.896310 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 47a67b1b9dc94e498cad91a72e655a75 (table default.Table_fa0e830600ba474bbbd6a68b9593d5f2 [id=16f2e0f0dbda4548871716a4a0f44dd4]) successfully deleted
I20251214 20:52:14.899341 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f198ef4dfa4f5791144b9a24fe693f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.727747 747 tablet_service.cc:1505] Processing CreateTablet for tablet 42bdcecba5534294aca1eb419f18e9d2 (DEFAULT_TABLE table=default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.903545 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bdcecba5534294aca1eb419f18e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.728097 752 tablet_service.cc:1505] Processing CreateTablet for tablet cbbfcef827ee4d548939aaf4783c8109 (DEFAULT_TABLE table=default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.909559 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbfcef827ee4d548939aaf4783c8109. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.728690 742 tablet_service.cc:1505] Processing CreateTablet for tablet ec85857f047e44d78835305ddcb70768 (DEFAULT_TABLE table=default.Table_58a7d5bd289f408da592c143483b0329 [id=bb060058712440c7b1e5c40332a0263d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.916498 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec85857f047e44d78835305ddcb70768. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.920320 1207 ts_tablet_manager.cc:1929] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:14.920642 1207 log.cc:1199] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ebedf28e17a741169e5490822a2d95df
I20251214 20:52:14.921157 1207 ts_tablet_manager.cc:1950] T ebedf28e17a741169e5490822a2d95df P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:14.922247 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 166) took 2333 ms. Trace:
I20251214 20:52:14.922459 1207 rpcz_store.cc:276] 1214 20:52:12.588307 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.588369 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:14.621251 (+2032882us) ts_tablet_manager.cc:1151] Deleting tablet ebedf28e17a741169e5490822a2d95df
1214 20:52:14.621262 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.915408 (+294146us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.920146 (+ 4738us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:14.922172 (+ 2026us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2029779}
W20251214 20:52:14.734367 712 consensus_peers.cc:597] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c17126f75b6241c5833269d20774c439. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:14.924050 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 189711us
I20251214 20:52:14.925474 743 tablet_service.cc:1552] Processing DeleteTablet for tablet ec85857f047e44d78835305ddcb70768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.925674 755 tablet_service.cc:1505] Processing CreateTablet for tablet ba7952290e2c4746802cffeb14971da0 (DEFAULT_TABLE table=default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.755573 756 tablet_service.cc:1505] Processing CreateTablet for tablet 738668adf2874d6894db6fa875f6990c (DEFAULT_TABLE table=default.Table_b92e50faaffb45c0a3e1c7a94166f985 [id=406157031d694ada8a9fcba4e4e23e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.925905 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 35a529702f924a5c9bf53c3c668c6094 with delete_type TABLET_DATA_DELETED (replaced by tablet 719d157ac8f74eb7911df05ab5e4a3d5) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.926147 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba7952290e2c4746802cffeb14971da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.926295 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.926621 754 tablet_service.cc:1552] Processing DeleteTablet for tablet a10663bd306d40e7a6375d06d69be22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.926855 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.927233 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 738668adf2874d6894db6fa875f6990c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.768136 746 tablet_service.cc:1505] Processing CreateTablet for tablet 98105bceda9b44bf893f82a4234a3c81 (DEFAULT_TABLE table=default.Table_e532e5fa17f2476a90df03eb6449a067 [id=ece0780edb0f41b787b46b290c4c0515]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.841872 1465 consensus_queue.cc:1048] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:14.851585 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "810ba69dda7f4631b1432bd99029609b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:14.853320 713 consensus_peers.cc:597] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 119c8cc0d6eb4d7f9a5d7fb3000c347b. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:14.931854 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 78602.1us
I20251214 20:52:14.933897 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_893229542bec44e8b903bb0934bd16f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.934798 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_893229542bec44e8b903bb0934bd16f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.936422 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_dd97baf658ea4f44bba79f081b5ff75e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.937029 524 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd97baf658ea4f44bba79f081b5ff75e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.733069 749 tablet_service.cc:1505] Processing CreateTablet for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 (DEFAULT_TABLE table=default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.925756 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 63ef97aeeaa94f4da683fbd133b05bd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b1a937150ce4ecfa210c8465d682496) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.937641 744 tablet_service.cc:1505] Processing CreateTablet for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (DEFAULT_TABLE table=default.Table_f487575ab59448afa92cbbb86139dd53 [id=52a7552cd4b44ca68f0ad5f1538a99a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.938270 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_934b554f3bb44b5e91119f9b78acb150" } new_table_name: "default.Table_1917701c01bf48c7b2bfc335d05f6db0" modify_external_catalogs: true
I20251214 20:52:14.923871 1476 raft_consensus.cc:493] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.940009 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 1dca369dc5f54a4ea1783e334c5670d6) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.940179 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98105bceda9b44bf893f82a4234a3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.941501 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 376e5efede87410eb9f95f36bedebf9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.941973 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658e4fe0d188483b9ceb9eb3b3f20835. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.942996 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_88881a27865d4b0aa6db6a2b736f0c17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.943595 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88881a27865d4b0aa6db6a2b736f0c17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.945236 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_3e8e22029bad4918b223e4c2fbf75c03" } new_table_name: "default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0" modify_external_catalogs: true
I20251214 20:52:14.947304 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ebedf28e17a741169e5490822a2d95df (table default.Table_506ad42d4328442997401fa286d1874d [id=08c79b0f6de64d6397fbc34f9aa9fc96]) successfully deleted
I20251214 20:52:14.947954 1476 raft_consensus.cc:515] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
W20251214 20:52:14.948282 1533 tablet_replica.cc:1410] Time spent flushing the Log queue took a long time: real 0.240s user 0.000s sys 0.000s
W20251214 20:52:14.948230 848 leader_election.cc:343] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.948419 848 leader_election.cc:304] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:14.948972 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:14.949390 848 raft_consensus.cc:2686] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251214 20:52:14.949707 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 65391.7us
I20251214 20:52:14.949869 1476 leader_election.cc:290] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:14.952772 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "810ba69dda7f4631b1432bd99029609b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:14.954558 1208 ts_tablet_manager.cc:1929] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:14.954900 1208 log.cc:1199] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/8e2ee5cc95304fc1a80ed4d31838ac38
I20251214 20:52:14.955293 1208 ts_tablet_manager.cc:1950] T 8e2ee5cc95304fc1a80ed4d31838ac38 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:14.957410 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_025651c33fe34ab1a55af7368f32fa04" } modify_external_catalogs: true
I20251214 20:52:14.957692 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_025651c33fe34ab1a55af7368f32fa04" } modify_external_catalogs: true
I20251214 20:52:14.960203 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ab99f8e2fad843e6966b1d86e41d58f6" } modify_external_catalogs: true
I20251214 20:52:14.960376 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_ab99f8e2fad843e6966b1d86e41d58f6" } modify_external_catalogs: true
I20251214 20:52:14.960938 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_005b061b84dc4e7aa3fe295da6b99cd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:14.961558 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_005b061b84dc4e7aa3fe295da6b99cd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:14.966750 1510 raft_consensus.cc:493] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:14.966923 1510 raft_consensus.cc:515] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:14.967065 772 raft_consensus.cc:2468] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:14.967458 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 167) took 2265 ms. Trace:
I20251214 20:52:14.967550 1207 rpcz_store.cc:276] 1214 20:52:12.682583 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.682664 (+ 81us) service_pool.cc:225] Handling call
1214 20:52:14.947877 (+2265213us) ts_tablet_manager.cc:1151] Deleting tablet 118dd44151a34ba3ae58f9fe1d3790d4
1214 20:52:14.947890 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.948039 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2264940}
I20251214 20:52:14.967684 1510 leader_election.cc:290] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:14.896363 1205 rpcz_store.cc:276] 1214 20:52:13.529597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.530032 (+ 435us) service_pool.cc:225] Handling call
1214 20:52:14.896026 (+1365994us) ts_tablet_manager.cc:1151] Deleting tablet a0279992d1f2419ab1caae116a0391b2
1214 20:52:14.896037 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.896215 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365759}
I20251214 20:52:14.968242 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 168) took 2273 ms. Trace:
I20251214 20:52:14.968327 1207 rpcz_store.cc:276] 1214 20:52:12.694383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.694462 (+ 79us) service_pool.cc:225] Handling call
1214 20:52:14.967802 (+2273340us) ts_tablet_manager.cc:1151] Deleting tablet c25832da5c1644329e18653000f3330e
1214 20:52:14.967811 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.967916 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2273079}
W20251214 20:52:14.968505 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a0279992d1f2419ab1caae116a0391b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0279992d1f2419ab1caae116a0391b2 already in progress: creating tablet
I20251214 20:52:14.969022 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 211) took 1386 ms. Trace:
I20251214 20:52:14.969148 1205 rpcz_store.cc:276] 1214 20:52:13.582600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.582662 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:14.968818 (+1386156us) ts_tablet_manager.cc:1151] Deleting tablet cd88d1e95ee04ef596863e1ae3687de0
1214 20:52:14.968832 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.968963 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1373416}
W20251214 20:52:14.969628 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet cd88d1e95ee04ef596863e1ae3687de0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd88d1e95ee04ef596863e1ae3687de0 already in progress: creating tablet
W20251214 20:52:14.969703 712 consensus_peers.cc:597] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8e09c13caca4a1c9018387972c9f082. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:14.970093 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 216) took 1272 ms. Trace:
I20251214 20:52:14.970180 1205 rpcz_store.cc:276] 1214 20:52:13.697956 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.698044 (+ 88us) service_pool.cc:225] Handling call
1214 20:52:14.969915 (+1271871us) ts_tablet_manager.cc:1151] Deleting tablet a34eb8fa3e34427495fa5985b7ba81bf
1214 20:52:14.969925 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.970043 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1225985}
I20251214 20:52:14.970559 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 221) took 1156 ms. Trace:
I20251214 20:52:14.970639 1205 rpcz_store.cc:276] 1214 20:52:13.813969 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.815855 (+ 1886us) service_pool.cc:225] Handling call
1214 20:52:14.970398 (+1154543us) ts_tablet_manager.cc:1151] Deleting tablet d6c0f445cd7e44769d79ad5566fd8d97
1214 20:52:14.970406 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:14.970500 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1143629}
W20251214 20:52:14.970729 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 118dd44151a34ba3ae58f9fe1d3790d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 118dd44151a34ba3ae58f9fe1d3790d4 already in progress: creating tablet
I20251214 20:52:14.971578 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "810ba69dda7f4631b1432bd99029609b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:14.971830 915 raft_consensus.cc:2468] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
W20251214 20:52:14.971874 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a34eb8fa3e34427495fa5985b7ba81bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a34eb8fa3e34427495fa5985b7ba81bf already in progress: creating tablet
W20251214 20:52:14.972121 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet d6c0f445cd7e44769d79ad5566fd8d97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6c0f445cd7e44769d79ad5566fd8d97 already in progress: creating tablet
I20251214 20:52:14.972244 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0d661a1bb034053a4634ff537eafc7d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:14.972934 851 leader_election.cc:343] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.974046 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 37a33cc9bb784f72956ac9f5493a0c3d (DEFAULT_TABLE table=default.Table_afb01d0257eb4d95bff481e5c900da2c [id=4efcd09629ed4717b4031e916f8b5749]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.974453 848 leader_election.cc:304] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:14.974824 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a33cc9bb784f72956ac9f5493a0c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.975014 1510 raft_consensus.cc:2804] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:14.975159 1510 raft_consensus.cc:493] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:14.975251 1510 raft_consensus.cc:3060] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:14.976500 880 tablet_service.cc:1505] Processing CreateTablet for tablet 37a33cc9bb784f72956ac9f5493a0c3d (DEFAULT_TABLE table=default.Table_afb01d0257eb4d95bff481e5c900da2c [id=4efcd09629ed4717b4031e916f8b5749]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.976969 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a33cc9bb784f72956ac9f5493a0c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.978169 1510 raft_consensus.cc:515] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:14.979626 882 tablet_service.cc:1505] Processing CreateTablet for tablet 0c973e0111804b8b869068bdcc5869bf (DEFAULT_TABLE table=default.Table_aec975307a944985bbcc63f028de7eda [id=8f35dde4f9204c799e62c4d656abfd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.979763 893 tablet_service.cc:1505] Processing CreateTablet for tablet 47e813373fbd482298312ead4f037c01 (DEFAULT_TABLE table=default.Table_a60363e46077464b968f9187a8c7a8fc [id=708908b1be53473f8c5f7e191128e7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.980131 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c973e0111804b8b869068bdcc5869bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.980374 878 tablet_service.cc:1505] Processing CreateTablet for tablet 17550eb0ee6b476f8a8c304451f678b2 (DEFAULT_TABLE table=default.Table_6f66a268b27e4f9b8d09597224149553 [id=d36ff09def9741f8b440f7bdbfdc21a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.980747 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17550eb0ee6b476f8a8c304451f678b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.981352 1510 leader_election.cc:290] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:14.983158 876 tablet_service.cc:1505] Processing CreateTablet for tablet 58ac06f852fc4abea5cb213926002ca1 (DEFAULT_TABLE table=default.Table_3369f89218b140e7a52da8221726f4e1 [id=e901e2802c854b4f829e59e00aa90040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:14.983592 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ac06f852fc4abea5cb213926002ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.984112 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e813373fbd482298312ead4f037c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:14.987998 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 8e2ee5cc95304fc1a80ed4d31838ac38 (table default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d]) successfully deleted
I20251214 20:52:14.988986 891 tablet_service.cc:1505] Processing CreateTablet for tablet 57f2314d33fa474ca36dafe0a0eae5ae (DEFAULT_TABLE table=default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:14.989641 713 leader_election.cc:343] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:14.991264 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 810ba69dda7f4631b1432bd99029609b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:14.996675 712 leader_election.cc:304] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:14.998106 1208 tablet_replica.cc:333] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:14.998342 1208 raft_consensus.cc:2243] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:14.998514 1203 tablet_bootstrap.cc:492] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:14.998543 1208 raft_consensus.cc:2272] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:14.998835 1203 ts_tablet_manager.cc:1403] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.579s user 0.000s sys 0.006s
I20251214 20:52:14.999027 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0d661a1bb034053a4634ff537eafc7d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:14.999142 1208 ts_tablet_manager.cc:1916] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.000030 1203 raft_consensus.cc:359] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.000097 753 tablet_service.cc:1505] Processing CreateTablet for tablet 47e813373fbd482298312ead4f037c01 (DEFAULT_TABLE table=default.Table_a60363e46077464b968f9187a8c7a8fc [id=708908b1be53473f8c5f7e191128e7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.000417 1203 raft_consensus.cc:385] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.000559 1203 raft_consensus.cc:740] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.000897 1203 consensus_queue.cc:260] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.001101 739 tablet_service.cc:1505] Processing CreateTablet for tablet 9b1a937150ce4ecfa210c8465d682496 (DEFAULT_TABLE table=default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.001335 1207 tablet_replica.cc:333] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:15.001453 745 tablet_service.cc:1505] Processing CreateTablet for tablet e90d11d69b8d4fd49150aa67a8190e5b (DEFAULT_TABLE table=default.Table_cf7fd4bd0b764cc79acc374ea7bb5187 [id=b3df2580f4a74189aa06f021d6b4ce71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.001725 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "810ba69dda7f4631b1432bd99029609b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:15.001947 1203 ts_tablet_manager.cc:1434] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:15.002265 1203 tablet_bootstrap.cc:492] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:15.002374 1207 raft_consensus.cc:2243] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.003783 742 tablet_service.cc:1505] Processing CreateTablet for tablet 37a33cc9bb784f72956ac9f5493a0c3d (DEFAULT_TABLE table=default.Table_afb01d0257eb4d95bff481e5c900da2c [id=4efcd09629ed4717b4031e916f8b5749]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.005088 1208 ts_tablet_manager.cc:1929] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.006541 751 tablet_service.cc:1505] Processing CreateTablet for tablet 57f2314d33fa474ca36dafe0a0eae5ae (DEFAULT_TABLE table=default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.007403 748 tablet_service.cc:1505] Processing CreateTablet for tablet 69f6108792fa4c2a9744e3248c76251f (DEFAULT_TABLE table=default.Table_b3f5ab40d9334de1a194aab3347387d7 [id=b470f1b24a6d4e7ab4420ae212e54ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.005249 1202 tablet_bootstrap.cc:492] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:15.007196 889 tablet_service.cc:1505] Processing CreateTablet for tablet 9b1a937150ce4ecfa210c8465d682496 (DEFAULT_TABLE table=default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.008077 892 tablet_service.cc:1505] Processing CreateTablet for tablet 69f6108792fa4c2a9744e3248c76251f (DEFAULT_TABLE table=default.Table_b3f5ab40d9334de1a194aab3347387d7 [id=b470f1b24a6d4e7ab4420ae212e54ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.008106 755 tablet_service.cc:1505] Processing CreateTablet for tablet 0c973e0111804b8b869068bdcc5869bf (DEFAULT_TABLE table=default.Table_aec975307a944985bbcc63f028de7eda [id=8f35dde4f9204c799e62c4d656abfd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.008512 884 tablet_service.cc:1505] Processing CreateTablet for tablet e90d11d69b8d4fd49150aa67a8190e5b (DEFAULT_TABLE table=default.Table_cf7fd4bd0b764cc79acc374ea7bb5187 [id=b3df2580f4a74189aa06f021d6b4ce71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.009130 754 tablet_service.cc:1505] Processing CreateTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 (DEFAULT_TABLE table=default.Table_63c9838172164e519ff318d7f570da85 [id=826a2666d167435c8fdfd9ab8b8819e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.009507 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e813373fbd482298312ead4f037c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.009709 883 tablet_service.cc:1505] Processing CreateTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 (DEFAULT_TABLE table=default.Table_63c9838172164e519ff318d7f570da85 [id=826a2666d167435c8fdfd9ab8b8819e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.009800 752 tablet_service.cc:1505] Processing CreateTablet for tablet 17550eb0ee6b476f8a8c304451f678b2 (DEFAULT_TABLE table=default.Table_6f66a268b27e4f9b8d09597224149553 [id=d36ff09def9741f8b440f7bdbfdc21a8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:15.010120 848 leader_election.cc:343] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:15.010911 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 69f6108792fa4c2a9744e3248c76251f (DEFAULT_TABLE table=default.Table_b3f5ab40d9334de1a194aab3347387d7 [id=b470f1b24a6d4e7ab4420ae212e54ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.011057 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f2314d33fa474ca36dafe0a0eae5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.011171 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 (DEFAULT_TABLE table=default.Table_63c9838172164e519ff318d7f570da85 [id=826a2666d167435c8fdfd9ab8b8819e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.011422 1031 tablet_service.cc:1505] Processing CreateTablet for tablet e90d11d69b8d4fd49150aa67a8190e5b (DEFAULT_TABLE table=default.Table_cf7fd4bd0b764cc79acc374ea7bb5187 [id=b3df2580f4a74189aa06f021d6b4ce71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.011482 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 0c973e0111804b8b869068bdcc5869bf (DEFAULT_TABLE table=default.Table_aec975307a944985bbcc63f028de7eda [id=8f35dde4f9204c799e62c4d656abfd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.011754 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f2314d33fa474ca36dafe0a0eae5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.011821 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 57f2314d33fa474ca36dafe0a0eae5ae (DEFAULT_TABLE table=default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.012138 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 9b1a937150ce4ecfa210c8465d682496 (DEFAULT_TABLE table=default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.012514 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 9da5e98c6f4843449886176e9fcfa8e1 (DEFAULT_TABLE table=default.Table_b0d8d299bd0b4950948be859b03a3731 [id=5ded9fe825ae4308b5bd4b21c16634c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.013156 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f6108792fa4c2a9744e3248c76251f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.013953 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "810ba69dda7f4631b1432bd99029609b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:15.014178 1055 raft_consensus.cc:3060] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:15.017014 776 raft_consensus.cc:3060] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:15.017725 1055 raft_consensus.cc:2468] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:15.018380 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98b7cac3930a4a9eb16ac413c8f1e092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.021183 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 17550eb0ee6b476f8a8c304451f678b2 (DEFAULT_TABLE table=default.Table_6f66a268b27e4f9b8d09597224149553 [id=d36ff09def9741f8b440f7bdbfdc21a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.021708 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17550eb0ee6b476f8a8c304451f678b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.021919 750 tablet_service.cc:1505] Processing CreateTablet for tablet 58ac06f852fc4abea5cb213926002ca1 (DEFAULT_TABLE table=default.Table_3369f89218b140e7a52da8221726f4e1 [id=e901e2802c854b4f829e59e00aa90040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.023911 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f2314d33fa474ca36dafe0a0eae5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.025096 1203 tablet_bootstrap.cc:654] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:15.027493 713 consensus_peers.cc:597] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8e09c13caca4a1c9018387972c9f082. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:15.028020 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b1a937150ce4ecfa210c8465d682496. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.031828 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c973e0111804b8b869068bdcc5869bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.031847 1202 ts_tablet_manager.cc:1403] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.222s user 0.013s sys 0.008s
I20251214 20:52:15.032660 848 leader_election.cc:304] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:15.034179 1208 log.cc:1199] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/810ba69dda7f4631b1432bd99029609b
I20251214 20:52:15.035486 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90d11d69b8d4fd49150aa67a8190e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.035791 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98b7cac3930a4a9eb16ac413c8f1e092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.037958 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da5e98c6f4843449886176e9fcfa8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.041745 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90d11d69b8d4fd49150aa67a8190e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.042025 776 raft_consensus.cc:2468] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:15.049211 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_7c95904dd8b044e4ad472681d5a5a274" } modify_external_catalogs: true
I20251214 20:52:15.049276 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4e95a2afe6004b6eb41370206c6faa1f" } modify_external_catalogs: true
I20251214 20:52:15.049430 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_7c95904dd8b044e4ad472681d5a5a274" } modify_external_catalogs: true
I20251214 20:52:15.049541 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_4e95a2afe6004b6eb41370206c6faa1f" } modify_external_catalogs: true
I20251214 20:52:15.050406 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98b7cac3930a4a9eb16ac413c8f1e092. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.050940 1510 raft_consensus.cc:2749] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:15.051959 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2342c4af4f9b420b9b7cbcf327ce1733" } modify_external_catalogs: true
I20251214 20:52:15.052173 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_2342c4af4f9b420b9b7cbcf327ce1733" } modify_external_catalogs: true
I20251214 20:52:15.052452 747 tablet_service.cc:1505] Processing CreateTablet for tablet 9da5e98c6f4843449886176e9fcfa8e1 (DEFAULT_TABLE table=default.Table_b0d8d299bd0b4950948be859b03a3731 [id=5ded9fe825ae4308b5bd4b21c16634c8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:15.053409 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd8226b431c6456885e928880938969d consensus state has no leader
I20251214 20:52:15.053736 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_1ae6ddcefec342c9a783873d39b77237"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.054260 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ae6ddcefec342c9a783873d39b77237 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.055581 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_74ac058b83ac4ff3944bce76a31834d6" } new_table_name: "default.Table_c488616dc265498088254d401a3cc7db" modify_external_catalogs: true
W20251214 20:52:15.056653 521 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca57963bcfb54eec9fcbed82757d8387 consensus state has no leader
I20251214 20:52:15.056951 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_f2360eda2fac4147840743a9a4f0bc2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.057390 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2360eda2fac4147840743a9a4f0bc2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.058528 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b1a937150ce4ecfa210c8465d682496. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.058601 1208 ts_tablet_manager.cc:1950] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:15.058710 848 leader_election.cc:304] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:15.059428 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 569a02142c71427d91868b02695ae12b consensus state has no leader
I20251214 20:52:15.059764 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_2ae9b9f412384e96bfa1e3e6d26ea104"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.060245 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ae9b9f412384e96bfa1e3e6d26ea104 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.060988 1208 tablet_replica.cc:333] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
W20251214 20:52:15.061283 848 raft_consensus.cc:2686] T 810ba69dda7f4631b1432bd99029609b P a2e8fd1cb731446d88f8e510c7a52999: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251214 20:52:15.058072 1476 raft_consensus.cc:2764] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251214 20:52:15.063201 1207 raft_consensus.cc:2272] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.063297 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_73af42b244d14d33889b754b78a9fd11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.063755 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73af42b244d14d33889b754b78a9fd11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.064467 1202 raft_consensus.cc:359] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.064713 1202 raft_consensus.cc:385] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.065137 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da5e98c6f4843449886176e9fcfa8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.065150 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f6108792fa4c2a9744e3248c76251f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.065629 1202 raft_consensus.cc:740] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.065891 1202 consensus_queue.cc:260] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.069576 1208 raft_consensus.cc:2243] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.069754 1208 raft_consensus.cc:2272] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:15.070402 712 consensus_peers.cc:597] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 119c8cc0d6eb4d7f9a5d7fb3000c347b. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:15.072430 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17550eb0ee6b476f8a8c304451f678b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.073274 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 3bbb28259c1442ae94f4eff76c25c539
I20251214 20:52:15.073697 1207 ts_tablet_manager.cc:1916] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.073961 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_afb01d0257eb4d95bff481e5c900da2c" } new_table_name: "default.Table_4a9b23e6d6a048d28ad16576ebf33c24" modify_external_catalogs: true
I20251214 20:52:15.074220 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f6108792fa4c2a9744e3248c76251f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.075228 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 8d9f74fb2c6244a6887e3c922262bbab
I20251214 20:52:15.075659 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b1a937150ce4ecfa210c8465d682496. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.075842 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_18ac89da14c642398e5f36730d316fee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.076340 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18ac89da14c642398e5f36730d316fee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:15.076844 712 consensus_peers.cc:597] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fd0fa0dad974c98bf6f52c8cfe83035. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:15.077276 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ac06f852fc4abea5cb213926002ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.079649 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90d11d69b8d4fd49150aa67a8190e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.083367 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c973e0111804b8b869068bdcc5869bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.086841 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 810ba69dda7f4631b1432bd99029609b (table default.Table_850408f19d2848309b88410e331f0798 [id=c544a159297d49c8be1c9a0eff9986cf]) successfully deleted
I20251214 20:52:15.071116 1208 ts_tablet_manager.cc:1916] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.071169 1533 tablet.cc:1722] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20251214 20:52:15.088279 518 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d164747cb294bd4b126b4420231867d consensus state has no leader
I20251214 20:52:15.073561 1202 ts_tablet_manager.cc:1434] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.036s user 0.000s sys 0.002s
I20251214 20:52:15.089370 518 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:15.091388 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet c2ae9dc8ead8403cb7215af8778872f0
I20251214 20:52:15.094520 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.096028 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.096321 746 tablet_service.cc:1552] Processing DeleteTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.097011 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet bb698ff30fcc47d09000cc68d2c5225b
I20251214 20:52:15.100348 1202 tablet_bootstrap.cc:492] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:15.102051 746 tablet_service.cc:1552] Processing DeleteTablet for tablet bb698ff30fcc47d09000cc68d2c5225b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.102365 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.102985 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37a33cc9bb784f72956ac9f5493a0c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.108937 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.126546 756 tablet_service.cc:1505] Processing CreateTablet for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (DEFAULT_TABLE table=default.Table_c845c10eeec54a7aa19cb8e0cebeb3a6 [id=4329d01e7f6147d0adad53fa86369b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.126638 876 tablet_service.cc:1552] Processing DeleteTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.127101 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 876d2a1f1d1c4f6882a5857b43568ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.127223 523 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:15.131570 892 tablet_service.cc:1505] Processing CreateTablet for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (DEFAULT_TABLE table=default.Table_c845c10eeec54a7aa19cb8e0cebeb3a6 [id=4329d01e7f6147d0adad53fa86369b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.132103 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 876d2a1f1d1c4f6882a5857b43568ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.139477 751 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab (DEFAULT_TABLE table=default.Table_025651c33fe34ab1a55af7368f32fa04 [id=e52f1b7d812d41cb80078ce966183aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.139966 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9f74fb2c6244a6887e3c922262bbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.140722 1202 tablet_bootstrap.cc:654] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:15.145653 522 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37a33cc9bb784f72956ac9f5493a0c3d consensus state has no leader
I20251214 20:52:15.146893 1206 tablet_bootstrap.cc:492] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:15.147732 876 tablet_service.cc:1505] Processing CreateTablet for tablet 9da5e98c6f4843449886176e9fcfa8e1 (DEFAULT_TABLE table=default.Table_b0d8d299bd0b4950948be859b03a3731 [id=5ded9fe825ae4308b5bd4b21c16634c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.149546 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da5e98c6f4843449886176e9fcfa8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.151392 884 tablet_service.cc:1552] Processing DeleteTablet for tablet bb698ff30fcc47d09000cc68d2c5225b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.160239 891 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab (DEFAULT_TABLE table=default.Table_025651c33fe34ab1a55af7368f32fa04 [id=e52f1b7d812d41cb80078ce966183aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.160873 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9f74fb2c6244a6887e3c922262bbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.163921 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet bb698ff30fcc47d09000cc68d2c5225b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.164109 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 58ac06f852fc4abea5cb213926002ca1 (DEFAULT_TABLE table=default.Table_3369f89218b140e7a52da8221726f4e1 [id=e901e2802c854b4f829e59e00aa90040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.164597 1208 ts_tablet_manager.cc:1929] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.164753 1208 log.cc:1199] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/6b4f582d8165439d86d9228c146dc41a
I20251214 20:52:15.165179 1208 ts_tablet_manager.cc:1950] T 6b4f582d8165439d86d9228c146dc41a P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:15.166646 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 6b4f582d8165439d86d9228c146dc41a (table default.Table_22df18c031ab4778ae90eb3580e9b8b7 [id=3f105cc6dd6944ceb9783a1ebe923d11]) successfully deleted
I20251214 20:52:15.166720 884 tablet_service.cc:1505] Processing CreateTablet for tablet 96dc4bdabb234c3fa4c26bd1633e783d (DEFAULT_TABLE table=default.Table_5e8a4451754247c2add9a3245b63b050 [id=d7f40373ddad4286a8fe7966df2d9668]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:15.166934 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 35a529702f924a5c9bf53c3c668c6094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a529702f924a5c9bf53c3c668c6094 already in progress: creating tablet
I20251214 20:52:15.167187 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96dc4bdabb234c3fa4c26bd1633e783d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.169508 1206 ts_tablet_manager.cc:1403] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.559s user 0.005s sys 0.001s
I20251214 20:52:15.173980 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 60ae3db968f6412aa0c3952fafe65f88 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:15.176004 754 tablet_service.cc:1505] Processing CreateTablet for tablet 96dc4bdabb234c3fa4c26bd1633e783d (DEFAULT_TABLE table=default.Table_5e8a4451754247c2add9a3245b63b050 [id=d7f40373ddad4286a8fe7966df2d9668]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.179347 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.181408 522 master_service.cc:954] ReplaceTablet: received request to replace tablet b315744c8cd043c5a8bdceec73a7e28b from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:15.186301 1513 raft_consensus.cc:493] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:15.186513 1513 raft_consensus.cc:515] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.187337 1513 leader_election.cc:290] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:15.189396 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 8af99e0bd8774a678575847a9ec2e557 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:15.189783 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.190121 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 47e813373fbd482298312ead4f037c01 (DEFAULT_TABLE table=default.Table_a60363e46077464b968f9187a8c7a8fc [id=708908b1be53473f8c5f7e191128e7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.190464 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.190454 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13f93bb5e5f4b59bcf9f4bfed070f25" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:15.190949 1206 raft_consensus.cc:359] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
W20251214 20:52:15.191010 848 leader_election.cc:343] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:15.191118 1206 raft_consensus.cc:385] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.191232 1206 raft_consensus.cc:740] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.191466 1206 consensus_queue.cc:260] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.192016 1206 ts_tablet_manager.cc:1434] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20251214 20:52:15.192139 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96dc4bdabb234c3fa4c26bd1633e783d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:15.193090 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3bbb28259c1442ae94f4eff76c25c539 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bbb28259c1442ae94f4eff76c25c539 already in progress: creating tablet
I20251214 20:52:15.193145 739 tablet_service.cc:1505] Processing CreateTablet for tablet 1dca369dc5f54a4ea1783e334c5670d6 (DEFAULT_TABLE table=default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.195353 876 tablet_service.cc:1505] Processing CreateTablet for tablet 1dca369dc5f54a4ea1783e334c5670d6 (DEFAULT_TABLE table=default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:15.195559 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8d9f74fb2c6244a6887e3c922262bbab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9f74fb2c6244a6887e3c922262bbab already in progress: creating tablet
I20251214 20:52:15.195724 525 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet f0d661a1bb034053a4634ff537eafc7d
I20251214 20:52:15.195777 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dca369dc5f54a4ea1783e334c5670d6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:15.197070 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c2ae9dc8ead8403cb7215af8778872f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2ae9dc8ead8403cb7215af8778872f0 already in progress: creating tablet
I20251214 20:52:15.198511 891 tablet_service.cc:1552] Processing DeleteTablet for tablet f0d661a1bb034053a4634ff537eafc7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:43856
W20251214 20:52:15.199090 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet bb698ff30fcc47d09000cc68d2c5225b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb698ff30fcc47d09000cc68d2c5225b already in progress: creating tablet
I20251214 20:52:15.200237 523 catalog_manager.cc:6199] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7] alter complete (version 4)
I20251214 20:52:15.201300 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ac06f852fc4abea5cb213926002ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.201576 1208 tablet_replica.cc:333] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:15.202193 523 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 4 got 0
I20251214 20:52:15.203817 1208 raft_consensus.cc:2243] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.204560 1207 ts_tablet_manager.cc:1929] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:15.204788 1207 log.cc:1199] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/8140dad1fab142d98b930b53ae5a64c2
I20251214 20:52:15.205190 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 224) took 1277 ms. Trace:
I20251214 20:52:15.205359 1207 ts_tablet_manager.cc:1950] T 8140dad1fab142d98b930b53ae5a64c2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:15.206635 884 tablet_service.cc:1505] Processing CreateTablet for tablet 453cf734f8b34b59aa246d618ffeade5 (DEFAULT_TABLE table=default.Table_3922bb23d5a34e7ab2e51a997856bffb [id=611e3a3e47ba4f91a705e9b8d3c323f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.205595 1208 raft_consensus.cc:2272] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.208216 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453cf734f8b34b59aa246d618ffeade5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.208592 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (DEFAULT_TABLE table=default.Table_c845c10eeec54a7aa19cb8e0cebeb3a6 [id=4329d01e7f6147d0adad53fa86369b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.209185 1205 rpcz_store.cc:276] 1214 20:52:13.881260 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.881402 (+ 142us) service_pool.cc:225] Handling call
1214 20:52:15.050020 (+1168618us) ts_tablet_manager.cc:1151] Deleting tablet ee7d709316b24b8eb223a838710c2979
1214 20:52:15.050038 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.158529 (+108491us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1159002}
I20251214 20:52:15.211349 752 tablet_service.cc:1505] Processing CreateTablet for tablet 453cf734f8b34b59aa246d618ffeade5 (DEFAULT_TABLE table=default.Table_3922bb23d5a34e7ab2e51a997856bffb [id=611e3a3e47ba4f91a705e9b8d3c323f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.213402 753 tablet_service.cc:1552] Processing DeleteTablet for tablet f0d661a1bb034053a4634ff537eafc7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.214519 1208 ts_tablet_manager.cc:1916] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.215749 1205 tablet_replica.cc:333] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:15.215994 1205 raft_consensus.cc:2243] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:15.216369 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e813373fbd482298312ead4f037c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.216949 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab (DEFAULT_TABLE table=default.Table_025651c33fe34ab1a55af7368f32fa04 [id=e52f1b7d812d41cb80078ce966183aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.221101 523 master_service.cc:954] ReplaceTablet: received request to replace tablet a10663bd306d40e7a6375d06d69be22d from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:15.225400 522 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet b315744c8cd043c5a8bdceec73a7e28b
I20251214 20:52:15.225889 522 catalog_manager.cc:6724] ReplaceTablet: tablet b315744c8cd043c5a8bdceec73a7e28b deleted and replaced by tablet f945814952b14131ba4e0c67a403a680
I20251214 20:52:15.227222 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dca369dc5f54a4ea1783e334c5670d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.228188 522 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 4 got 0
I20251214 20:52:15.228220 891 tablet_service.cc:1552] Processing DeleteTablet for tablet b315744c8cd043c5a8bdceec73a7e28b with delete_type TABLET_DATA_DELETED (replaced by tablet f945814952b14131ba4e0c67a403a680) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.233999 753 tablet_service.cc:1552] Processing DeleteTablet for tablet b315744c8cd043c5a8bdceec73a7e28b with delete_type TABLET_DATA_DELETED (replaced by tablet f945814952b14131ba4e0c67a403a680) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.235725 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 169) took 2479 ms. Trace:
I20251214 20:52:15.235930 1207 rpcz_store.cc:276] 1214 20:52:12.756542 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.756636 (+ 94us) service_pool.cc:225] Handling call
1214 20:52:14.968565 (+2211929us) ts_tablet_manager.cc:1151] Deleting tablet 8140dad1fab142d98b930b53ae5a64c2
1214 20:52:14.968572 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.156335 (+187763us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.204417 (+ 48082us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.235691 (+ 31274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2211598}
I20251214 20:52:15.236449 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 8140dad1fab142d98b930b53ae5a64c2 (table default.Table_d6e5176452664864914a6f76f32af5ac [id=3606c9fa9a3d4b55b86dddf293442a3c]) successfully deleted
I20251214 20:52:15.237032 1207 tablet_replica.cc:333] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:15.237296 1207 raft_consensus.cc:2243] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:15.238037 1207 pending_rounds.cc:70] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:15.238549 1207 pending_rounds.cc:77] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493702762401792 op_type: NO_OP noop_request { }
I20251214 20:52:15.238785 1207 raft_consensus.cc:2889] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:15.238957 1207 raft_consensus.cc:2272] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.239578 1207 ts_tablet_manager.cc:1916] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.243789 1206 tablet_bootstrap.cc:492] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:15.246198 1206 tablet_bootstrap.cc:654] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.247881 522 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 4 got 0
I20251214 20:52:15.249070 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 96dc4bdabb234c3fa4c26bd1633e783d (DEFAULT_TABLE table=default.Table_5e8a4451754247c2add9a3245b63b050 [id=d7f40373ddad4286a8fe7966df2d9668]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.249583 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9f74fb2c6244a6887e3c922262bbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.250826 1205 pending_rounds.cc:70] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:15.250903 1205 pending_rounds.cc:77] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493709230235648 op_type: NO_OP noop_request { }
I20251214 20:52:15.251586 1205 raft_consensus.cc:2889] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:15.251715 1205 raft_consensus.cc:2272] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.251053 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13f93bb5e5f4b59bcf9f4bfed070f25" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:15.254906 1205 ts_tablet_manager.cc:1916] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.257172 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 8140dad1fab142d98b930b53ae5a64c2 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:15.258424 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 1dca369dc5f54a4ea1783e334c5670d6 (DEFAULT_TABLE table=default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:15.260684 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ee7d709316b24b8eb223a838710c2979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7d709316b24b8eb223a838710c2979 already in progress: creating tablet
I20251214 20:52:15.261004 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_6a61fe3bf1734a888048db0d20abac60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.261415 851 leader_election.cc:343] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:15.261653 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a61fe3bf1734a888048db0d20abac60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.261634 851 leader_election.cc:304] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:15.262646 1513 raft_consensus.cc:2749] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:15.262681 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_00bef63ea36c4a7585c2832be1454b68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.263192 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00bef63ea36c4a7585c2832be1454b68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.263312 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_384f071b4b4d411aa4fe7bcd8160a32a" } new_table_name: "default.Table_ab93057983a146d488d93fb5b803be4f" modify_external_catalogs: true
I20251214 20:52:15.264793 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_f8f22fe9676e44c8931f7069ee213892" } new_table_name: "default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e" modify_external_catalogs: true
I20251214 20:52:15.265589 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96dc4bdabb234c3fa4c26bd1633e783d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.266002 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_a40d3fbe67ba492d96c7db49c759836e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.266423 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a40d3fbe67ba492d96c7db49c759836e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.267087 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet f0d661a1bb034053a4634ff537eafc7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.267325 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3d23e6dc5242452693b8aaac124a863f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.267709 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d23e6dc5242452693b8aaac124a863f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.268682 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_f487575ab59448afa92cbbb86139dd53" } new_table_name: "default.Table_ae63b32352284405b5918f5b33f80e5f" modify_external_catalogs: true
I20251214 20:52:15.269721 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_ee7f67ff1a1c4014a1cc4f9cf7f6613b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.270103 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee7f67ff1a1c4014a1cc4f9cf7f6613b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.271275 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet b315744c8cd043c5a8bdceec73a7e28b with delete_type TABLET_DATA_DELETED (replaced by tablet f945814952b14131ba4e0c67a403a680) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.271714 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dca369dc5f54a4ea1783e334c5670d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.275815 1513 raft_consensus.cc:493] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:15.276017 1513 raft_consensus.cc:515] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.276834 1513 leader_election.cc:290] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:15.278045 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "144dee1a6d7344f2b79da6e79e67599e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:15.278896 848 leader_election.cc:343] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:15.277912 1208 ts_tablet_manager.cc:1929] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.279563 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_caa5b0dabd9d4ea8a8b16c550407dc53" } modify_external_catalogs: true
I20251214 20:52:15.279748 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_caa5b0dabd9d4ea8a8b16c550407dc53" } modify_external_catalogs: true
I20251214 20:52:15.279495 1208 log.cc:1199] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/f0d661a1bb034053a4634ff537eafc7d
I20251214 20:52:15.280870 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453cf734f8b34b59aa246d618ffeade5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.280958 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a60363e46077464b968f9187a8c7a8fc" } modify_external_catalogs: true
I20251214 20:52:15.281111 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_a60363e46077464b968f9187a8c7a8fc" } modify_external_catalogs: true
I20251214 20:52:15.281911 1208 ts_tablet_manager.cc:1950] T f0d661a1bb034053a4634ff537eafc7d P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:15.287729 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "144dee1a6d7344f2b79da6e79e67599e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:15.287855 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet f0d661a1bb034053a4634ff537eafc7d (table default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90]) successfully deleted
W20251214 20:52:15.288362 851 leader_election.cc:343] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:15.288534 851 leader_election.cc:304] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:15.288990 1513 raft_consensus.cc:2749] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:15.289800 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet b315744c8cd043c5a8bdceec73a7e28b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b315744c8cd043c5a8bdceec73a7e28b already in progress: creating tablet
I20251214 20:52:15.290006 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 876d2a1f1d1c4f6882a5857b43568ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.295521 1565 tablet.cc:1722] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251214 20:52:15.301822 1202 tablet_bootstrap.cc:492] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:15.302172 1202 ts_tablet_manager.cc:1403] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.213s user 0.004s sys 0.025s
I20251214 20:52:15.303496 1202 raft_consensus.cc:359] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.303851 1202 raft_consensus.cc:385] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.303984 1202 raft_consensus.cc:740] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.304394 1202 consensus_queue.cc:260] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.306246 1202 ts_tablet_manager.cc:1434] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20251214 20:52:15.308845 1202 tablet_bootstrap.cc:492] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:15.314531 1202 tablet_bootstrap.cc:654] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.316249 1206 tablet_bootstrap.cc:492] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:15.316488 1206 ts_tablet_manager.cc:1403] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.000s
I20251214 20:52:15.317323 1206 raft_consensus.cc:359] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.317639 1206 raft_consensus.cc:385] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.317756 1206 raft_consensus.cc:740] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.318095 1206 consensus_queue.cc:260] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.320926 1206 ts_tablet_manager.cc:1434] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251214 20:52:15.321281 1206 tablet_bootstrap.cc:492] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:15.323004 1206 tablet_bootstrap.cc:654] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.329420 1207 ts_tablet_manager.cc:1929] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:15.330363 1207 log.cc:1199] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ecb69e385bf34709b5d954de4b660087
I20251214 20:52:15.330215 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 453cf734f8b34b59aa246d618ffeade5 (DEFAULT_TABLE table=default.Table_3922bb23d5a34e7ab2e51a997856bffb [id=611e3a3e47ba4f91a705e9b8d3c323f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.330845 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453cf734f8b34b59aa246d618ffeade5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.331169 1207 ts_tablet_manager.cc:1950] T ecb69e385bf34709b5d954de4b660087 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:15.334846 1202 tablet_bootstrap.cc:492] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:15.335649 1202 ts_tablet_manager.cc:1403] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
W20251214 20:52:15.336589 517 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d6e7306254044449f1127fb963ca5f0 consensus state has no leader
W20251214 20:52:15.336838 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 658e4fe0d188483b9ceb9eb3b3f20835 consensus state has no leader
I20251214 20:52:15.338742 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_3922bb23d5a34e7ab2e51a997856bffb" } new_table_name: "default.Table_1de37433cd074bd494bcaa2064ea1823" modify_external_catalogs: true
I20251214 20:52:15.339576 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_18ac89da14c642398e5f36730d316fee" } modify_external_catalogs: true
I20251214 20:52:15.339737 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_18ac89da14c642398e5f36730d316fee" } modify_external_catalogs: true
I20251214 20:52:15.339738 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 63497e77baeb4d328be2421b5732a28a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.340092 1208 tablet_replica.cc:333] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:15.340310 1208 raft_consensus.cc:2243] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.340490 1208 raft_consensus.cc:2272] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.340595 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_250b3222a83c44b08856444499fa612d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.341449 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_250b3222a83c44b08856444499fa612d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.341629 1208 ts_tablet_manager.cc:1916] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.342129 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_7b427a00182f4e059423bf0e18859b90" } modify_external_catalogs: true
I20251214 20:52:15.342303 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_7b427a00182f4e059423bf0e18859b90" } modify_external_catalogs: true
I20251214 20:52:15.340022 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_f57172cc26534c9eafbefb6eb9a4d2e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.343197 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:15.344143 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f4760585baf4b4ab3b9b94b5a748a61 consensus state has no leader
I20251214 20:52:15.344205 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 170) took 2587 ms. Trace:
I20251214 20:52:15.344307 1207 rpcz_store.cc:276] 1214 20:52:12.756668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.756729 (+ 61us) service_pool.cc:225] Handling call
1214 20:52:15.236971 (+2480242us) ts_tablet_manager.cc:1151] Deleting tablet ecb69e385bf34709b5d954de4b660087
1214 20:52:15.236981 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.304283 (+ 67302us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.329246 (+ 24963us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.344167 (+ 14921us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2479893}
I20251214 20:52:15.344470 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_55706259657e4a09aa794297547e9d98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:15.344837 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 171) took 2549 ms. Trace:
I20251214 20:52:15.344940 1207 rpcz_store.cc:276] 1214 20:52:12.795524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.795639 (+ 115us) service_pool.cc:225] Handling call
1214 20:52:15.344688 (+2549049us) ts_tablet_manager.cc:1151] Deleting tablet 12b5f4326aa0413eb303c132ba21decd
1214 20:52:15.344696 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.344805 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2548659}
W20251214 20:52:15.345013 526 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55706259657e4a09aa794297547e9d98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.345367 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 172) took 2549 ms. Trace:
I20251214 20:52:15.345460 1207 rpcz_store.cc:276] 1214 20:52:12.795682 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.796090 (+ 408us) service_pool.cc:225] Handling call
1214 20:52:15.345258 (+2549168us) ts_tablet_manager.cc:1151] Deleting tablet 84a0237e4ee34f94821d8647a400575b
1214 20:52:15.345266 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.345339 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2548836}
I20251214 20:52:15.345813 1207 tablet_replica.cc:333] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:15.345992 1207 raft_consensus.cc:2243] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:15.346269 1207 pending_rounds.cc:70] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:15.346830 1207 pending_rounds.cc:77] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493706387611648 op_type: NO_OP noop_request { }
I20251214 20:52:15.346997 1207 raft_consensus.cc:2889] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:15.347179 1207 raft_consensus.cc:2272] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.349009 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_13e60d7ade27439bbcab22ee125a50ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.349577 523 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13e60d7ade27439bbcab22ee125a50ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.350124 1208 ts_tablet_manager.cc:1929] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.350479 1208 log.cc:1199] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/63497e77baeb4d328be2421b5732a28a
I20251214 20:52:15.350919 1206 tablet_bootstrap.cc:492] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:15.351085 1208 ts_tablet_manager.cc:1950] T 63497e77baeb4d328be2421b5732a28a P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:15.351161 1206 ts_tablet_manager.cc:1403] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20251214 20:52:15.352541 1206 raft_consensus.cc:359] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.352757 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 63497e77baeb4d328be2421b5732a28a (table default.Table_d553ca568da242899d768f9848baba3d [id=376e4eabd04d4a53ab2772323b8c41b7]) successfully deleted
W20251214 20:52:15.352854 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet bb698ff30fcc47d09000cc68d2c5225b (table default.Table_7c95904dd8b044e4ad472681d5a5a274 [id=8c96acfc06ea4f00996fa066d719e418], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bb698ff30fcc47d09000cc68d2c5225b consensus state has no leader
I20251214 20:52:15.353011 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ecb69e385bf34709b5d954de4b660087 (table default.Table_500b29027c11444893ae5b7a3123cb8b [id=49474ea35419441c9218bc8c896ffd34]) successfully deleted
I20251214 20:52:15.353013 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 118dd44151a34ba3ae58f9fe1d3790d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.353185 1206 raft_consensus.cc:385] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251214 20:52:15.353266 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 12b5f4326aa0413eb303c132ba21decd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12b5f4326aa0413eb303c132ba21decd already in progress: creating tablet
I20251214 20:52:15.353327 1206 raft_consensus.cc:740] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
W20251214 20:52:15.353485 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 84a0237e4ee34f94821d8647a400575b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84a0237e4ee34f94821d8647a400575b already in progress: creating tablet
I20251214 20:52:15.353696 1206 consensus_queue.cc:260] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.353663 1202 raft_consensus.cc:359] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.354305 1202 raft_consensus.cc:385] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.354514 1202 raft_consensus.cc:740] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.355062 1202 consensus_queue.cc:260] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.360144 1207 ts_tablet_manager.cc:1916] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.362038 1202 ts_tablet_manager.cc:1434] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20251214 20:52:15.360328 1206 ts_tablet_manager.cc:1434] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:15.362589 1206 tablet_bootstrap.cc:492] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:15.363862 1202 tablet_bootstrap.cc:492] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:15.366194 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 12b5f4326aa0413eb303c132ba21decd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.368327 884 tablet_service.cc:1552] Processing DeleteTablet for tablet c25832da5c1644329e18653000f3330e with delete_type TABLET_DATA_DELETED (replaced by tablet 0c26d1f969ed4071a2f2cf1615a17cac) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.368625 1208 tablet_replica.cc:333] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:15.368849 1208 raft_consensus.cc:2243] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.369062 1208 raft_consensus.cc:2272] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.370172 1208 ts_tablet_manager.cc:1916] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.370442 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_0dcbd217695546df9d1ca32583980907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.371042 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dcbd217695546df9d1ca32583980907 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:15.373818 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 120553us
I20251214 20:52:15.374202 1203 tablet_bootstrap.cc:492] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:15.374351 1476 raft_consensus.cc:493] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:15.374385 1203 ts_tablet_manager.cc:1403] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.372s user 0.014s sys 0.034s
I20251214 20:52:15.374519 1476 raft_consensus.cc:515] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.375212 1205 ts_tablet_manager.cc:1929] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:15.375393 1476 leader_election.cc:290] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:15.375319 1203 raft_consensus.cc:359] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.376377 1205 log.cc:1199] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ed1d007f1ca24b7291262708ddfca3c3
I20251214 20:52:15.376346 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "118dd44151a34ba3ae58f9fe1d3790d4" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:15.376459 1203 raft_consensus.cc:385] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.376652 1203 raft_consensus.cc:740] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:15.376829 712 leader_election.cc:343] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 118dd44151a34ba3ae58f9fe1d3790d4
I20251214 20:52:15.376924 1203 consensus_queue.cc:260] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.377648 1206 tablet_bootstrap.cc:654] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.379088 1055 raft_consensus.cc:2468] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:15.380142 1203 ts_tablet_manager.cc:1434] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:15.380507 1205 ts_tablet_manager.cc:1950] T ed1d007f1ca24b7291262708ddfca3c3 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:15.380788 1203 tablet_bootstrap.cc:492] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:15.381669 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 7fcfcfe81fe24d369eea696efc61d21c
I20251214 20:52:15.379859 713 leader_election.cc:304] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:15.383567 1476 raft_consensus.cc:2804] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:15.383566 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_7f018aa7a0aa4ab3895001f6c97c48bd" } new_table_name: "default.Table_7abff44151df46dd935898812837e8b8" modify_external_catalogs: true
I20251214 20:52:15.383667 1476 raft_consensus.cc:493] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:15.383760 1476 raft_consensus.cc:3060] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:15.385454 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.385829 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 225) took 1495 ms. Trace:
I20251214 20:52:15.385974 1205 rpcz_store.cc:276] 1214 20:52:13.886008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.890100 (+ 4092us) service_pool.cc:225] Handling call
1214 20:52:15.212521 (+1322421us) ts_tablet_manager.cc:1151] Deleting tablet ed1d007f1ca24b7291262708ddfca3c3
1214 20:52:15.212536 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.285522 (+ 72986us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.375042 (+ 89520us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.381497 (+ 6455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1318846}
I20251214 20:52:15.386276 1476 raft_consensus.cc:515] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.386551 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ed1d007f1ca24b7291262708ddfca3c3 (table default.Table_e3603e6d524c40dea2a1a88b79d63a8e [id=7febb36ade704d57bdfd4211054277ab]) successfully deleted
I20251214 20:52:15.386864 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.388236 1476 leader_election.cc:290] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:15.388536 712 leader_election.cc:343] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 118dd44151a34ba3ae58f9fe1d3790d4
I20251214 20:52:15.390744 1202 tablet_bootstrap.cc:654] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.393409 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.396391 1208 ts_tablet_manager.cc:1929] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.396731 1208 log.cc:1199] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/c25832da5c1644329e18653000f3330e
I20251214 20:52:15.397378 1208 ts_tablet_manager.cc:1950] T c25832da5c1644329e18653000f3330e P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:15.399509 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet c25832da5c1644329e18653000f3330e (table default.Table_c5d8c1bdf3344ec5a0d409a1abf98b45 [id=5427e484a5a94314a13be3f387c06d64]) successfully deleted
W20251214 20:52:15.399835 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 7fcfcfe81fe24d369eea696efc61d21c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fcfcfe81fe24d369eea696efc61d21c already in progress: creating tablet
I20251214 20:52:15.400210 1207 ts_tablet_manager.cc:1929] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:15.400323 1207 log.cc:1199] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/c17126f75b6241c5833269d20774c439
I20251214 20:52:15.400697 1207 ts_tablet_manager.cc:1950] T c17126f75b6241c5833269d20774c439 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:15.401427 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 173) took 2554 ms. Trace:
I20251214 20:52:15.403801 1202 tablet_bootstrap.cc:492] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:15.404094 1202 ts_tablet_manager.cc:1403] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.002s
I20251214 20:52:15.406008 1202 raft_consensus.cc:359] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.406953 1202 raft_consensus.cc:385] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.407115 1202 raft_consensus.cc:740] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.407618 1202 consensus_queue.cc:260] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.410734 520 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 4 got 0
I20251214 20:52:15.411548 1202 ts_tablet_manager.cc:1434] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251214 20:52:15.413843 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1bba0cb42a0c4aa986d55fdc492ffd46" } modify_external_catalogs: true
I20251214 20:52:15.414078 884 tablet_service.cc:1552] Processing DeleteTablet for tablet b1df0dad2fb040a5a12e0401e00c87db with delete_type TABLET_DATA_DELETED (replaced by tablet 8fcdd6bafa5c4aa6b59668cde523d010) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.414206 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1bba0cb42a0c4aa986d55fdc492ffd46" } modify_external_catalogs: true
I20251214 20:52:15.415040 1208 tablet_replica.cc:333] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:15.415501 1208 raft_consensus.cc:2243] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.415802 1208 raft_consensus.cc:2272] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.416675 1208 ts_tablet_manager.cc:1916] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.418615 776 raft_consensus.cc:3060] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:15.423506 1202 tablet_bootstrap.cc:492] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:15.427040 1202 tablet_bootstrap.cc:654] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.428761 1206 tablet_bootstrap.cc:492] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:15.429026 1206 ts_tablet_manager.cc:1403] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20251214 20:52:15.429870 1206 raft_consensus.cc:359] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.430112 1206 raft_consensus.cc:385] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.430228 1206 raft_consensus.cc:740] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
W20251214 20:52:15.431946 712 consensus_peers.cc:597] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc442726ec0c456290ecf2892d0df957. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:15.432111 1206 consensus_queue.cc:260] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.432732 1206 ts_tablet_manager.cc:1434] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:15.433104 1206 tablet_bootstrap.cc:492] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:15.434136 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "118dd44151a34ba3ae58f9fe1d3790d4" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:15.436549 1206 tablet_bootstrap.cc:654] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.401531 1207 rpcz_store.cc:276] 1214 20:52:12.846678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:12.846749 (+ 71us) service_pool.cc:225] Handling call
1214 20:52:15.345744 (+2498995us) ts_tablet_manager.cc:1151] Deleting tablet c17126f75b6241c5833269d20774c439
1214 20:52:15.345752 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.372335 (+ 26583us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.400059 (+ 27724us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.401391 (+ 1332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2498651}
I20251214 20:52:15.439118 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet c17126f75b6241c5833269d20774c439 (table default.Table_834f1d323b2b4785a85f667e5c14e739 [id=587a162643cf4bc5a3b4c9fc0de83340]) successfully deleted
I20251214 20:52:15.438804 1208 ts_tablet_manager.cc:1929] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.439678 1207 tablet_replica.cc:333] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:15.439579 1208 log.cc:1199] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/b1df0dad2fb040a5a12e0401e00c87db
I20251214 20:52:15.440006 1207 raft_consensus.cc:2243] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:15.440351 1207 pending_rounds.cc:70] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:15.440469 1207 pending_rounds.cc:77] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493706302615552 op_type: NO_OP noop_request { }
I20251214 20:52:15.440613 1207 raft_consensus.cc:2889] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:15.440779 1207 raft_consensus.cc:2272] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.441980 1208 ts_tablet_manager.cc:1950] T b1df0dad2fb040a5a12e0401e00c87db P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:15.443085 1207 ts_tablet_manager.cc:1916] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.445080 1055 raft_consensus.cc:3060] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:15.446055 1203 tablet_bootstrap.cc:654] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.451948 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 810ba69dda7f4631b1432bd99029609b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.454496 1055 raft_consensus.cc:2468] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:15.454747 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 2a7fc15882e2474bb6885070be13b1fa from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:15.455889 713 leader_election.cc:304] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:15.456413 1476 raft_consensus.cc:2804] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:15.456686 1476 raft_consensus.cc:697] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:15.456735 1533 tablet.cc:1717] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: Skipping requested alter to schema version 4, tablet already version 4
I20251214 20:52:15.457022 1476 consensus_queue.cc:237] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.466032 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet b1df0dad2fb040a5a12e0401e00c87db (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]) successfully deleted
I20251214 20:52:15.468880 517 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 4 got 0
I20251214 20:52:15.469545 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 12b5f4326aa0413eb303c132ba21decd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.475214 1576 tablet.cc:1722] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251214 20:52:15.479755 1206 tablet_bootstrap.cc:492] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:15.479939 1206 ts_tablet_manager.cc:1403] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20251214 20:52:15.480684 1206 raft_consensus.cc:359] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.480870 1206 raft_consensus.cc:385] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.480952 1206 raft_consensus.cc:740] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.478920 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 227) took 1482 ms. Trace:
I20251214 20:52:15.481257 1206 consensus_queue.cc:260] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.481782 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 3e5e220d083243eb8aa4328f366cd7ff from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:15.481834 1206 ts_tablet_manager.cc:1434] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:15.482134 1206 tablet_bootstrap.cc:492] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:15.483814 1206 tablet_bootstrap.cc:654] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:15.486475 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 453cf734f8b34b59aa246d618ffeade5 consensus state has no leader
I20251214 20:52:15.487035 1205 rpcz_store.cc:276] 1214 20:52:13.903546 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.905578 (+ 2032us) service_pool.cc:225] Handling call
1214 20:52:15.386305 (+1480727us) ts_tablet_manager.cc:1151] Deleting tablet 40a9a6192c744282ab5672cbe15fd299
1214 20:52:15.386315 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.386453 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479722}
W20251214 20:52:15.490736 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 40a9a6192c744282ab5672cbe15fd299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40a9a6192c744282ab5672cbe15fd299 already in progress: creating tablet
I20251214 20:52:15.491509 1207 ts_tablet_manager.cc:1929] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:15.491741 1207 log.cc:1199] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/119c8cc0d6eb4d7f9a5d7fb3000c347b
I20251214 20:52:15.492272 1207 ts_tablet_manager.cc:1950] T 119c8cc0d6eb4d7f9a5d7fb3000c347b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:15.492777 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 182) took 2079 ms. Trace:
I20251214 20:52:15.495586 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 228) took 1470 ms. Trace:
I20251214 20:52:15.501935 1207 rpcz_store.cc:276] 1214 20:52:13.412961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.413022 (+ 61us) service_pool.cc:225] Handling call
1214 20:52:15.439605 (+2026583us) ts_tablet_manager.cc:1151] Deleting tablet 119c8cc0d6eb4d7f9a5d7fb3000c347b
1214 20:52:15.439614 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.458660 (+ 19046us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.491347 (+ 32687us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.492726 (+ 1379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2026241}
I20251214 20:52:15.502900 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 186) took 2043 ms. Trace:
I20251214 20:52:15.504776 1207 rpcz_store.cc:276] 1214 20:52:13.459761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.459824 (+ 63us) service_pool.cc:225] Handling call
1214 20:52:15.502687 (+2042863us) ts_tablet_manager.cc:1151] Deleting tablet 8af99e0bd8774a678575847a9ec2e557
1214 20:52:15.502700 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.502853 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2042492}
W20251214 20:52:15.505384 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8af99e0bd8774a678575847a9ec2e557 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8af99e0bd8774a678575847a9ec2e557 already in progress: creating tablet
I20251214 20:52:15.505929 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 185) took 2046 ms. Trace:
I20251214 20:52:15.508191 1565 tablet.cc:1717] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: Skipping requested alter to schema version 4, tablet already version 4
I20251214 20:52:15.513912 1207 rpcz_store.cc:276] 1214 20:52:13.459667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.460037 (+ 370us) service_pool.cc:225] Handling call
1214 20:52:15.505713 (+2045676us) ts_tablet_manager.cc:1151] Deleting tablet b652a8a80246443491551e76e68d2a8b
1214 20:52:15.505725 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.505886 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2045425}
I20251214 20:52:15.497192 1205 rpcz_store.cc:276] 1214 20:52:14.023998 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.024063 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:15.492141 (+1468078us) ts_tablet_manager.cc:1151] Deleting tablet a063083f54ad49d2a842fa080d44ed85
1214 20:52:15.492384 (+ 243us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.494500 (+ 2116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1467565}
W20251214 20:52:15.515159 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b652a8a80246443491551e76e68d2a8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b652a8a80246443491551e76e68d2a8b already in progress: creating tablet
I20251214 20:52:15.515590 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 84a0237e4ee34f94821d8647a400575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37374
W20251214 20:52:15.516222 524 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf04f91e79754569a51d468ab576b0dc consensus state has no leader
I20251214 20:52:15.516521 1533 tablet.cc:1717] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: Skipping requested alter to schema version 4, tablet already version 4
W20251214 20:52:15.518224 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a063083f54ad49d2a842fa080d44ed85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a063083f54ad49d2a842fa080d44ed85 already in progress: creating tablet
I20251214 20:52:15.521342 1202 tablet_bootstrap.cc:492] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:15.522317 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 230) took 1460 ms. Trace:
I20251214 20:52:15.522485 1205 rpcz_store.cc:276] 1214 20:52:14.056060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.056126 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:15.516343 (+1460217us) ts_tablet_manager.cc:1151] Deleting tablet b0e805cc899347a083d4d549ad68e2f9
1214 20:52:15.516353 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.516775 (+ 422us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1459894}
I20251214 20:52:15.522959 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 233) took 1309 ms. Trace:
I20251214 20:52:15.523051 1205 rpcz_store.cc:276] 1214 20:52:14.213659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.213723 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:15.522785 (+1309062us) ts_tablet_manager.cc:1151] Deleting tablet ebedf28e17a741169e5490822a2d95df
1214 20:52:15.522797 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.522905 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1308787}
W20251214 20:52:15.523095 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b0e805cc899347a083d4d549ad68e2f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0e805cc899347a083d4d549ad68e2f9 already in progress: creating tablet
I20251214 20:52:15.523429 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 239) took 1264 ms. Trace:
W20251214 20:52:15.523733 513 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ebedf28e17a741169e5490822a2d95df because the tablet was not found. No further retry: Not found: Tablet not found: ebedf28e17a741169e5490822a2d95df
I20251214 20:52:15.523957 1202 ts_tablet_manager.cc:1403] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.099s user 0.025s sys 0.021s
W20251214 20:52:15.524063 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 46547.7us
I20251214 20:52:15.525700 1202 raft_consensus.cc:359] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.526182 1565 tablet.cc:1717] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: Skipping requested alter to schema version 4, tablet already version 4
I20251214 20:52:15.528501 1576 tablet.cc:1717] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: Skipping requested alter to schema version 4, tablet already version 4
I20251214 20:52:15.531114 1202 raft_consensus.cc:385] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.531358 1202 raft_consensus.cc:740] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.533015 1202 consensus_queue.cc:260] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.538630 1202 ts_tablet_manager.cc:1434] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251214 20:52:15.545928 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 189) took 2059 ms. Trace:
I20251214 20:52:15.546531 884 tablet_service.cc:1505] Processing CreateTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 (DEFAULT_TABLE table=default.Table_18ac89da14c642398e5f36730d316fee [id=56cb939dcaf243a389548a1d68d52081]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.547008 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d949abef1cb41a9905482d81a98d2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.548840 1202 tablet_bootstrap.cc:492] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:15.550817 1202 tablet_bootstrap.cc:654] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.523520 1205 rpcz_store.cc:276] 1214 20:52:14.258718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.258839 (+ 121us) service_pool.cc:225] Handling call
1214 20:52:15.523277 (+1264438us) ts_tablet_manager.cc:1151] Deleting tablet 324379e3217442c3addcfe58060b9ad7
1214 20:52:15.523284 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.523380 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264074}
W20251214 20:52:15.552151 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 324379e3217442c3addcfe58060b9ad7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 324379e3217442c3addcfe58060b9ad7 already in progress: creating tablet
I20251214 20:52:15.556226 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 240) took 1297 ms. Trace:
I20251214 20:52:15.556389 1205 rpcz_store.cc:276] 1214 20:52:14.259050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.259103 (+ 53us) service_pool.cc:225] Handling call
1214 20:52:15.555992 (+1296889us) ts_tablet_manager.cc:1151] Deleting tablet 24ff58f321124d12b5330dca185d426e
1214 20:52:15.556008 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.556166 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1296574}
I20251214 20:52:15.556931 1205 tablet_replica.cc:333] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:15.556907 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 24ff58f321124d12b5330dca185d426e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ff58f321124d12b5330dca185d426e already in progress: creating tablet
I20251214 20:52:15.557174 1205 raft_consensus.cc:2243] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.557439 1205 raft_consensus.cc:2272] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.558357 1207 rpcz_store.cc:276] 1214 20:52:13.477679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.477741 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:15.515500 (+2037759us) ts_tablet_manager.cc:1151] Deleting tablet 2da66ec83e9140f7a605f0c86abff468
1214 20:52:15.517196 (+ 1696us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.537348 (+ 20152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2037411}
I20251214 20:52:15.558733 1203 tablet_bootstrap.cc:492] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
W20251214 20:52:15.559041 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2da66ec83e9140f7a605f0c86abff468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2da66ec83e9140f7a605f0c86abff468 already in progress: creating tablet
I20251214 20:52:15.559224 1203 ts_tablet_manager.cc:1403] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.178s user 0.009s sys 0.003s
I20251214 20:52:15.559902 884 tablet_service.cc:1505] Processing CreateTablet for tablet 25dcf491653147e7831217c1ec7a2e0c (DEFAULT_TABLE table=default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.559887 754 tablet_service.cc:1505] Processing CreateTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 (DEFAULT_TABLE table=default.Table_18ac89da14c642398e5f36730d316fee [id=56cb939dcaf243a389548a1d68d52081]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.560107 1203 raft_consensus.cc:359] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.560294 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25dcf491653147e7831217c1ec7a2e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.560343 1203 raft_consensus.cc:385] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.560418 1203 raft_consensus.cc:740] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.560647 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 193) took 2045 ms. Trace:
I20251214 20:52:15.560724 1203 consensus_queue.cc:260] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.562780 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_63c9838172164e519ff318d7f570da85" } new_table_name: "default.Table_e5f3db562155448e91e2e4314254cbe6" modify_external_catalogs: true
I20251214 20:52:15.565284 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_180ac2f089ba44298206726a47570c5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.565814 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_180ac2f089ba44298206726a47570c5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.567445 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_1ae6ddcefec342c9a783873d39b77237" } new_table_name: "default.Table_7bc0f81d22a643589e8a54ceb2477807" modify_external_catalogs: true
I20251214 20:52:15.567612 1576 tablet.cc:1717] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: Skipping requested alter to schema version 4, tablet already version 4
I20251214 20:52:15.568573 1203 ts_tablet_manager.cc:1434] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251214 20:52:15.568626 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d949abef1cb41a9905482d81a98d2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.569204 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.569538 1208 tablet_replica.cc:333] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:15.569747 1208 raft_consensus.cc:2243] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.569948 1208 raft_consensus.cc:2272] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.570325 1203 tablet_bootstrap.cc:492] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:15.571362 1208 ts_tablet_manager.cc:1916] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.572274 1203 tablet_bootstrap.cc:654] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.579452 526 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 2a7fc15882e2474bb6885070be13b1fa
I20251214 20:52:15.579972 526 catalog_manager.cc:6724] ReplaceTablet: tablet 2a7fc15882e2474bb6885070be13b1fa deleted and replaced by tablet 65e79db74fa046698507c83416ee1669
I20251214 20:52:15.581423 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a7fc15882e2474bb6885070be13b1fa with delete_type TABLET_DATA_DELETED (replaced by tablet 65e79db74fa046698507c83416ee1669) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.581656 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a7fc15882e2474bb6885070be13b1fa with delete_type TABLET_DATA_DELETED (replaced by tablet 65e79db74fa046698507c83416ee1669) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.583494 1476 consensus_queue.cc:1048] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:15.584664 1208 ts_tablet_manager.cc:1929] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.585013 1208 log.cc:1199] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/8faa7fd7315f485c82aee38bcf4cd8fc
I20251214 20:52:15.585539 1208 ts_tablet_manager.cc:1950] T 8faa7fd7315f485c82aee38bcf4cd8fc P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:15.586882 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 8faa7fd7315f485c82aee38bcf4cd8fc (table default.Table_1d198a3975c24b84aaeb82c266c746b8 [id=bc62e2586be140efbc2bfb5a3b5baaa2]) successfully deleted
W20251214 20:52:15.588248 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2a7fc15882e2474bb6885070be13b1fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a7fc15882e2474bb6885070be13b1fa already in progress: creating tablet
I20251214 20:52:15.589470 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_aec975307a944985bbcc63f028de7eda" } new_table_name: "default.Table_d8953672be144941ab3319e5477b8c57" modify_external_catalogs: true
I20251214 20:52:15.589826 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.590574 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a7fc15882e2474bb6885070be13b1fa with delete_type TABLET_DATA_DELETED (replaced by tablet 65e79db74fa046698507c83416ee1669) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.590659 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet b0e805cc899347a083d4d549ad68e2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:15.591032 512 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8faa7fd7315f485c82aee38bcf4cd8fc because the tablet was not found. No further retry: Not found: Tablet not found: 8faa7fd7315f485c82aee38bcf4cd8fc
I20251214 20:52:15.590953 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 (DEFAULT_TABLE table=default.Table_18ac89da14c642398e5f36730d316fee [id=56cb939dcaf243a389548a1d68d52081]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.591475 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d949abef1cb41a9905482d81a98d2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.560938 1207 rpcz_store.cc:276] 1214 20:52:13.515026 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.515180 (+ 154us) service_pool.cc:225] Handling call
1214 20:52:15.560338 (+2045158us) ts_tablet_manager.cc:1151] Deleting tablet 52daf1204bf142e7b82852bdec6582fb
1214 20:52:15.560346 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.560501 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2044845}
W20251214 20:52:15.591863 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 52daf1204bf142e7b82852bdec6582fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52daf1204bf142e7b82852bdec6582fb already in progress: creating tablet
I20251214 20:52:15.592516 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 194) took 2077 ms. Trace:
I20251214 20:52:15.592978 1207 rpcz_store.cc:276] 1214 20:52:13.515170 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.515520 (+ 350us) service_pool.cc:225] Handling call
1214 20:52:15.592370 (+2076850us) ts_tablet_manager.cc:1151] Deleting tablet 604b7c6fe3b24daca70711a1990af556
1214 20:52:15.592381 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.592484 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2076686}
I20251214 20:52:15.593834 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 25dcf491653147e7831217c1ec7a2e0c (DEFAULT_TABLE table=default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.593904 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 197) took 2077 ms. Trace:
I20251214 20:52:15.594007 1207 rpcz_store.cc:276] 1214 20:52:13.518937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.519421 (+ 484us) service_pool.cc:225] Handling call
1214 20:52:15.593753 (+2074332us) ts_tablet_manager.cc:1151] Deleting tablet 472e4a5990e64d02b1d52dc6e86d7190
1214 20:52:15.593762 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.593870 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2068288}
W20251214 20:52:15.594096 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 604b7c6fe3b24daca70711a1990af556 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 604b7c6fe3b24daca70711a1990af556 already in progress: creating tablet
I20251214 20:52:15.594254 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25dcf491653147e7831217c1ec7a2e0c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:15.595214 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 472e4a5990e64d02b1d52dc6e86d7190 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 472e4a5990e64d02b1d52dc6e86d7190 already in progress: creating tablet
I20251214 20:52:15.594594 754 tablet_service.cc:1505] Processing CreateTablet for tablet 25dcf491653147e7831217c1ec7a2e0c (DEFAULT_TABLE table=default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.595760 892 tablet_service.cc:1505] Processing CreateTablet for tablet 06381efe75284e649fd6d1a9903cfa9b (DEFAULT_TABLE table=default.Table_1ae6ddcefec342c9a783873d39b77237 [id=db33edd24660463ea648be7da745e29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.595818 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 06381efe75284e649fd6d1a9903cfa9b (DEFAULT_TABLE table=default.Table_1ae6ddcefec342c9a783873d39b77237 [id=db33edd24660463ea648be7da745e29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.596145 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06381efe75284e649fd6d1a9903cfa9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.596172 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06381efe75284e649fd6d1a9903cfa9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.595955 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25dcf491653147e7831217c1ec7a2e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.598886 742 tablet_service.cc:1505] Processing CreateTablet for tablet 06381efe75284e649fd6d1a9903cfa9b (DEFAULT_TABLE table=default.Table_1ae6ddcefec342c9a783873d39b77237 [id=db33edd24660463ea648be7da745e29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.599392 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06381efe75284e649fd6d1a9903cfa9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.600325 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 192) took 2085 ms. Trace:
I20251214 20:52:15.601595 1207 rpcz_store.cc:276] 1214 20:52:13.514907 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.515093 (+ 186us) service_pool.cc:225] Handling call
1214 20:52:15.600182 (+2085089us) ts_tablet_manager.cc:1151] Deleting tablet 3c3962c33a294fa98f8bd485268aee71
1214 20:52:15.600191 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.600286 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2074352}
I20251214 20:52:15.602447 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 200) took 1894 ms. Trace:
I20251214 20:52:15.603516 1207 rpcz_store.cc:276] 1214 20:52:13.708166 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.713505 (+ 5339us) service_pool.cc:225] Handling call
1214 20:52:15.602299 (+1888794us) ts_tablet_manager.cc:1151] Deleting tablet a0279992d1f2419ab1caae116a0391b2
1214 20:52:15.602308 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.602411 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1878976}
I20251214 20:52:15.603694 1205 ts_tablet_manager.cc:1916] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.604229 1207 tablet_replica.cc:333] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:15.604905 1207 raft_consensus.cc:2243] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:15.605744 1207 pending_rounds.cc:70] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:15.606451 1207 pending_rounds.cc:77] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493706278301696 op_type: NO_OP noop_request { }
I20251214 20:52:15.606909 1207 raft_consensus.cc:2889] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:15.607260 1207 raft_consensus.cc:2272] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:15.607244 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3c3962c33a294fa98f8bd485268aee71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c3962c33a294fa98f8bd485268aee71 already in progress: creating tablet
W20251214 20:52:15.607484 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet a0279992d1f2419ab1caae116a0391b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0279992d1f2419ab1caae116a0391b2 already in progress: creating tablet
I20251214 20:52:15.608165 1207 ts_tablet_manager.cc:1916] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.610096 521 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 3d949abef1cb41a9905482d81a98d2c1
I20251214 20:52:15.610850 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_26795454cb054bd6bdb4100765bd3c17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.611496 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26795454cb054bd6bdb4100765bd3c17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.614809 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.615038 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.615669 1206 tablet_bootstrap.cc:492] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:15.616472 517 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 3e5e220d083243eb8aa4328f366cd7ff
I20251214 20:52:15.616907 517 catalog_manager.cc:6724] ReplaceTablet: tablet 3e5e220d083243eb8aa4328f366cd7ff deleted and replaced by tablet 6a000951f61948ac85fdd431f3d162c2
I20251214 20:52:15.617234 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_59405cb9326245918d9cf6fb65ddbf40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.618402 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59405cb9326245918d9cf6fb65ddbf40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.618682 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff with delete_type TABLET_DATA_DELETED (replaced by tablet 6a000951f61948ac85fdd431f3d162c2) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.618817 1206 ts_tablet_manager.cc:1403] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.137s user 0.012s sys 0.003s
W20251214 20:52:15.619586 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3d949abef1cb41a9905482d81a98d2c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d949abef1cb41a9905482d81a98d2c1 already in progress: creating tablet
I20251214 20:52:15.619673 1206 raft_consensus.cc:359] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.619872 1206 raft_consensus.cc:385] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.619911 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff with delete_type TABLET_DATA_DELETED (replaced by tablet 6a000951f61948ac85fdd431f3d162c2) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.619951 1206 raft_consensus.cc:740] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.620261 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff with delete_type TABLET_DATA_DELETED (replaced by tablet 6a000951f61948ac85fdd431f3d162c2) from {username='slave'} at 127.0.0.1:37374
W20251214 20:52:15.620659 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3e5e220d083243eb8aa4328f366cd7ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e5e220d083243eb8aa4328f366cd7ff already in progress: creating tablet
I20251214 20:52:15.622044 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.625339 1206 consensus_queue.cc:260] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.626428 892 tablet_service.cc:1505] Processing CreateTablet for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (DEFAULT_TABLE table=default.Table_2ae9b9f412384e96bfa1e3e6d26ea104 [id=9553b46856c94003a293135c814a7df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.626885 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6bdfbe4d15345de8be5de00bfe4ca07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.628298 1206 ts_tablet_manager.cc:1434] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:15.632283 1206 tablet_bootstrap.cc:492] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:15.635521 1029 tablet_service.cc:1505] Processing CreateTablet for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (DEFAULT_TABLE table=default.Table_2ae9b9f412384e96bfa1e3e6d26ea104 [id=9553b46856c94003a293135c814a7df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.636796 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6bdfbe4d15345de8be5de00bfe4ca07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.637661 1206 tablet_bootstrap.cc:654] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.638386 1202 tablet_bootstrap.cc:492] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:15.638624 1202 ts_tablet_manager.cc:1403] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.009s
I20251214 20:52:15.639582 1202 raft_consensus.cc:359] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.639882 1202 raft_consensus.cc:385] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.639998 1202 raft_consensus.cc:740] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.640343 1202 consensus_queue.cc:260] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.640969 1202 ts_tablet_manager.cc:1434] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:15.641402 1202 tablet_bootstrap.cc:492] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:15.644130 518 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 47e813373fbd482298312ead4f037c01
I20251214 20:52:15.646560 1202 tablet_bootstrap.cc:654] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.648445 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_e82b745b92174d32a0d7bc8f779a6558"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:15.649022 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:15.649071 518 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e82b745b92174d32a0d7bc8f779a6558 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.650436 1203 tablet_bootstrap.cc:492] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:15.652084 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 12b5f4326aa0413eb303c132ba21decd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.652395 1208 tablet_replica.cc:333] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:15.652854 1208 raft_consensus.cc:2243] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.653052 1208 raft_consensus.cc:2272] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.653985 1208 ts_tablet_manager.cc:1916] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.651106 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.653100 742 tablet_service.cc:1505] Processing CreateTablet for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (DEFAULT_TABLE table=default.Table_2ae9b9f412384e96bfa1e3e6d26ea104 [id=9553b46856c94003a293135c814a7df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.659278 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6bdfbe4d15345de8be5de00bfe4ca07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.660686 1208 ts_tablet_manager.cc:1929] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.660912 1208 log.cc:1199] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/12b5f4326aa0413eb303c132ba21decd
I20251214 20:52:15.661383 1208 ts_tablet_manager.cc:1950] T 12b5f4326aa0413eb303c132ba21decd P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:15.661775 1203 ts_tablet_manager.cc:1403] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.093s user 0.000s sys 0.004s
I20251214 20:52:15.662952 1203 raft_consensus.cc:359] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.663961 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 12b5f4326aa0413eb303c132ba21decd (table default.Table_1437581b4e464656a47dc3ab99914c22 [id=b5ef4fb9aeba4c3fb423322b5f1d0e14]) successfully deleted
I20251214 20:52:15.664371 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:43856
W20251214 20:52:15.665107 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 47e813373fbd482298312ead4f037c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e813373fbd482298312ead4f037c01 already in progress: creating tablet
I20251214 20:52:15.666363 892 tablet_service.cc:1505] Processing CreateTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 (DEFAULT_TABLE table=default.Table_44bb60b4a8ec45329622c851c90b5aa8 [id=1fb24606cfcd4445800f51b30b7f0fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.666929 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84e9dfc23d34eb0a379e8ac5a442e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.672451 1206 tablet_bootstrap.cc:492] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:15.672657 1206 ts_tablet_manager.cc:1403] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
I20251214 20:52:15.673444 525 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 569a02142c71427d91868b02695ae12b
I20251214 20:52:15.673497 1206 raft_consensus.cc:359] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.674140 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 569a02142c71427d91868b02695ae12b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.673652 1206 raft_consensus.cc:385] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.674332 1206 raft_consensus.cc:740] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.674669 1206 consensus_queue.cc:260] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.675374 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_6a0720013c0d470e91997a7ad8d483fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.675933 525 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a0720013c0d470e91997a7ad8d483fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.677948 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_44bb60b4a8ec45329622c851c90b5aa8" } modify_external_catalogs: true
I20251214 20:52:15.678135 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_44bb60b4a8ec45329622c851c90b5aa8" } modify_external_catalogs: true
I20251214 20:52:15.679495 1206 ts_tablet_manager.cc:1434] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:15.679653 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 569a02142c71427d91868b02695ae12b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.680030 1208 tablet_replica.cc:333] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:15.680366 1208 raft_consensus.cc:2243] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.680645 1208 raft_consensus.cc:2272] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.681331 1208 ts_tablet_manager.cc:1916] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.681634 876 tablet_service.cc:1505] Processing CreateTablet for tablet 3a23561b9f4e48bbbd796e14fafec086 (DEFAULT_TABLE table=default.Table_f2360eda2fac4147840743a9a4f0bc2b [id=f1338e81f4ec4dce87130f2ddbe98041]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.682096 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a23561b9f4e48bbbd796e14fafec086. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.682969 742 tablet_service.cc:1505] Processing CreateTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 (DEFAULT_TABLE table=default.Table_44bb60b4a8ec45329622c851c90b5aa8 [id=1fb24606cfcd4445800f51b30b7f0fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.683228 1203 raft_consensus.cc:385] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.683370 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84e9dfc23d34eb0a379e8ac5a442e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.683478 1203 raft_consensus.cc:740] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.685554 1203 consensus_queue.cc:260] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.685806 1513 raft_consensus.cc:493] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:15.685950 1513 raft_consensus.cc:515] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.686632 1513 leader_election.cc:290] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:15.687321 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25974e5db555447baf5799775d15cd14" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:15.687425 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25974e5db555447baf5799775d15cd14" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:15.687803 851 leader_election.cc:343] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:15.688400 848 leader_election.cc:343] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:15.688553 1206 tablet_bootstrap.cc:492] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:15.688532 848 leader_election.cc:304] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:15.689481 1513 raft_consensus.cc:2749] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:15.689810 712 consensus_peers.cc:597] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 376e5efede87410eb9f95f36bedebf9d. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:15.692346 1206 tablet_bootstrap.cc:654] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.696552 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 3a23561b9f4e48bbbd796e14fafec086 (DEFAULT_TABLE table=default.Table_f2360eda2fac4147840743a9a4f0bc2b [id=f1338e81f4ec4dce87130f2ddbe98041]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.697069 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a23561b9f4e48bbbd796e14fafec086. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.653119 1207 ts_tablet_manager.cc:1929] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:15.698843 1207 log.cc:1199] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/a8e09c13caca4a1c9018387972c9f082
I20251214 20:52:15.699321 1207 ts_tablet_manager.cc:1950] T a8e09c13caca4a1c9018387972c9f082 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:15.700111 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 202) took 1963 ms. Trace:
I20251214 20:52:15.700237 1207 rpcz_store.cc:276] 1214 20:52:13.737232 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.737304 (+ 72us) service_pool.cc:225] Handling call
1214 20:52:15.604159 (+1866855us) ts_tablet_manager.cc:1151] Deleting tablet a8e09c13caca4a1c9018387972c9f082
1214 20:52:15.604167 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.637840 (+ 33673us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.652982 (+ 15142us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.700077 (+ 47095us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1858435}
I20251214 20:52:15.700688 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 203) took 1957 ms. Trace:
I20251214 20:52:15.700802 1207 rpcz_store.cc:276] 1214 20:52:13.750307 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.750504 (+ 197us) service_pool.cc:225] Handling call
1214 20:52:15.700548 (+1950044us) ts_tablet_manager.cc:1151] Deleting tablet cd88d1e95ee04ef596863e1ae3687de0
1214 20:52:15.700557 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.700662 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1949762}
I20251214 20:52:15.701088 520 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet bf8643c7e4094f529cd55656c7b3edb6
W20251214 20:52:15.701197 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 44493.3us
I20251214 20:52:15.701964 1208 ts_tablet_manager.cc:1929] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.702775 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.703037 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet a8e09c13caca4a1c9018387972c9f082 (table default.Table_4a73de3f8186423abc9e9123ed3f1acb [id=397be7c374504dc5973e1cbf6b263084]) successfully deleted
W20251214 20:52:15.703306 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet cd88d1e95ee04ef596863e1ae3687de0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd88d1e95ee04ef596863e1ae3687de0 already in progress: creating tablet
I20251214 20:52:15.703747 1208 log.cc:1199] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/569a02142c71427d91868b02695ae12b
I20251214 20:52:15.705538 884 tablet_service.cc:1552] Processing DeleteTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.705840 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 209) took 1863 ms. Trace:
I20251214 20:52:15.705978 1207 rpcz_store.cc:276] 1214 20:52:13.840262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.840370 (+ 108us) service_pool.cc:225] Handling call
1214 20:52:15.701034 (+1860664us) ts_tablet_manager.cc:1151] Deleting tablet a34eb8fa3e34427495fa5985b7ba81bf
1214 20:52:15.701042 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.703756 (+ 2714us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1825362}
I20251214 20:52:15.706350 1208 ts_tablet_manager.cc:1950] T 569a02142c71427d91868b02695ae12b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
W20251214 20:52:15.706384 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet a34eb8fa3e34427495fa5985b7ba81bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a34eb8fa3e34427495fa5985b7ba81bf already in progress: creating tablet
I20251214 20:52:15.707377 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 214) took 1837 ms. Trace:
I20251214 20:52:15.708441 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 569a02142c71427d91868b02695ae12b (table default.Table_7b427a00182f4e059423bf0e18859b90 [id=17c89c03be10405f834504b803e594c7]) successfully deleted
I20251214 20:52:15.708614 1207 rpcz_store.cc:276] 1214 20:52:13.869695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.869766 (+ 71us) service_pool.cc:225] Handling call
1214 20:52:15.707240 (+1837474us) ts_tablet_manager.cc:1151] Deleting tablet d6c0f445cd7e44769d79ad5566fd8d97
1214 20:52:15.707251 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.707348 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1830875}
W20251214 20:52:15.709215 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet d6c0f445cd7e44769d79ad5566fd8d97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6c0f445cd7e44769d79ad5566fd8d97 already in progress: creating tablet
W20251214 20:52:15.710036 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet bf8643c7e4094f529cd55656c7b3edb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf8643c7e4094f529cd55656c7b3edb6 already in progress: creating tablet
I20251214 20:52:15.711020 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cfd9b2f9756433e9e52643417c4cd45) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.710939 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_379fdcc125644411b3d7172ec3d66698"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:15.711480 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 218) took 1816 ms. Trace:
I20251214 20:52:15.711632 1207 rpcz_store.cc:276] 1214 20:52:13.895043 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.895174 (+ 131us) service_pool.cc:225] Handling call
1214 20:52:15.711333 (+1816159us) ts_tablet_manager.cc:1151] Deleting tablet ee7d709316b24b8eb223a838710c2979
1214 20:52:15.711343 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.711441 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815862}
I20251214 20:52:15.711931 1202 tablet_bootstrap.cc:492] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
W20251214 20:52:15.712136 520 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_379fdcc125644411b3d7172ec3d66698 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.712173 1202 ts_tablet_manager.cc:1403] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.006s
W20251214 20:52:15.712107 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ee7d709316b24b8eb223a838710c2979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7d709316b24b8eb223a838710c2979 already in progress: creating tablet
I20251214 20:52:15.712541 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 219) took 1806 ms. Trace:
I20251214 20:52:15.713680 1202 raft_consensus.cc:359] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.714010 1202 raft_consensus.cc:385] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.714174 1202 raft_consensus.cc:740] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.714493 1202 consensus_queue.cc:260] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.715101 1202 ts_tablet_manager.cc:1434] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:15.715003 1207 rpcz_store.cc:276] 1214 20:52:13.906099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.906161 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:15.712411 (+1806250us) ts_tablet_manager.cc:1151] Deleting tablet 40a9a6192c744282ab5672cbe15fd299
1214 20:52:15.712420 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.712503 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806013}
I20251214 20:52:15.715288 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e2ee5cc95304fc1a80ed4d31838ac38 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cfd9b2f9756433e9e52643417c4cd45) from {username='slave'} at 127.0.0.1:60078
W20251214 20:52:15.715430 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 40a9a6192c744282ab5672cbe15fd299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40a9a6192c744282ab5672cbe15fd299 already in progress: creating tablet
I20251214 20:52:15.715729 1202 tablet_bootstrap.cc:492] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:15.717857 1203 ts_tablet_manager.cc:1434] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.056s user 0.000s sys 0.002s
I20251214 20:52:15.721627 1203 tablet_bootstrap.cc:492] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:15.722885 1587 consensus_queue.cc:1048] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:15.723649 1203 tablet_bootstrap.cc:654] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.723659 1202 tablet_bootstrap.cc:654] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.670339 1205 ts_tablet_manager.cc:1929] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:15.678117 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 569a02142c71427d91868b02695ae12b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.679220 1526 raft_consensus.cc:493] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:15.730427 1526 raft_consensus.cc:515] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.730455 754 tablet_service.cc:1505] Processing CreateTablet for tablet 3a23561b9f4e48bbbd796e14fafec086 (DEFAULT_TABLE table=default.Table_f2360eda2fac4147840743a9a4f0bc2b [id=f1338e81f4ec4dce87130f2ddbe98041]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.730906 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a23561b9f4e48bbbd796e14fafec086. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.731261 1526 leader_election.cc:290] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:15.682049 1026 tablet_service.cc:1505] Processing CreateTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 (DEFAULT_TABLE table=default.Table_44bb60b4a8ec45329622c851c90b5aa8 [id=1fb24606cfcd4445800f51b30b7f0fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.731673 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84e9dfc23d34eb0a379e8ac5a442e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.731750 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0e805cc899347a083d4d549ad68e2f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:15.731989 772 raft_consensus.cc:2468] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:15.732450 989 leader_election.cc:304] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
W20251214 20:52:15.736285 991 leader_election.cc:343] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: b0e805cc899347a083d4d549ad68e2f9
I20251214 20:52:15.736778 1526 raft_consensus.cc:2804] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:15.736892 1526 raft_consensus.cc:493] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:15.736995 1526 raft_consensus.cc:3060] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:15.738703 519 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98b7cac3930a4a9eb16ac413c8f1e092 consensus state has no leader
I20251214 20:52:15.740078 1526 raft_consensus.cc:515] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
W20251214 20:52:15.741549 991 leader_election.cc:343] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: b0e805cc899347a083d4d549ad68e2f9
I20251214 20:52:15.741910 1526 leader_election.cc:290] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:15.742650 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_b5f628ed069a4b19996c9b579d7678bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:15.742779 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0e805cc899347a083d4d549ad68e2f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:15.742971 772 raft_consensus.cc:3060] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:15.745709 519 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5f628ed069a4b19996c9b579d7678bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.746150 772 raft_consensus.cc:2468] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:15.746697 989 leader_election.cc:304] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:15.747012 1526 raft_consensus.cc:2804] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:15.747215 1526 raft_consensus.cc:697] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:15.748188 1526 consensus_queue.cc:237] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.749011 884 tablet_service.cc:1505] Processing CreateTablet for tablet 492770a7b8b74ba49315f974cc54eedb (DEFAULT_TABLE table=default.Table_005b061b84dc4e7aa3fe295da6b99cd1 [id=d6c5c9fbf30b4477b502d001a4f3cd27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.749352 1205 log.cc:1199] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/3fd0fa0dad974c98bf6f52c8cfe83035
I20251214 20:52:15.749742 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 492770a7b8b74ba49315f974cc54eedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.749838 1205 ts_tablet_manager.cc:1950] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:15.750631 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 244) took 1475 ms. Trace:
I20251214 20:52:15.750762 1205 rpcz_store.cc:276] 1214 20:52:14.275490 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.275544 (+ 54us) service_pool.cc:225] Handling call
1214 20:52:15.556843 (+1281299us) ts_tablet_manager.cc:1151] Deleting tablet 3fd0fa0dad974c98bf6f52c8cfe83035
1214 20:52:15.556852 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.667043 (+110191us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.670204 (+ 3161us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.750596 (+ 80392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1281053}
I20251214 20:52:15.751232 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 492770a7b8b74ba49315f974cc54eedb (DEFAULT_TABLE table=default.Table_005b061b84dc4e7aa3fe295da6b99cd1 [id=d6c5c9fbf30b4477b502d001a4f3cd27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.751366 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 3fd0fa0dad974c98bf6f52c8cfe83035 (table default.Table_0173f37106d54db18ef3975bc6b0d185 [id=5c14810dd5364dcaa218b80c3756eadc]) successfully deleted
I20251214 20:52:15.751684 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 492770a7b8b74ba49315f974cc54eedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.755043 1513 raft_consensus.cc:493] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:15.755259 1513 raft_consensus.cc:515] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.755941 1513 leader_election.cc:290] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:15.756943 848 leader_election.cc:343] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 6381cbf05a8c4f84b71f229497fd28ef
W20251214 20:52:15.757228 851 leader_election.cc:343] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 6381cbf05a8c4f84b71f229497fd28ef
I20251214 20:52:15.757361 851 leader_election.cc:304] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:15.757915 1513 raft_consensus.cc:2749] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:15.762367 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_f1a942f776f74d61922b38c19e43f5df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.762959 522 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1a942f776f74d61922b38c19e43f5df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.766865 754 tablet_service.cc:1505] Processing CreateTablet for tablet 492770a7b8b74ba49315f974cc54eedb (DEFAULT_TABLE table=default.Table_005b061b84dc4e7aa3fe295da6b99cd1 [id=d6c5c9fbf30b4477b502d001a4f3cd27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.767357 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 492770a7b8b74ba49315f974cc54eedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.775166 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 220) took 1806 ms. Trace:
I20251214 20:52:15.775295 1207 rpcz_store.cc:276] 1214 20:52:13.968751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.971140 (+ 2389us) service_pool.cc:225] Handling call
1214 20:52:15.774935 (+1803795us) ts_tablet_manager.cc:1151] Deleting tablet a063083f54ad49d2a842fa080d44ed85
1214 20:52:15.774945 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.775091 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1803166}
I20251214 20:52:15.775600 1207 tablet_replica.cc:333] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:15.775787 1207 raft_consensus.cc:2243] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:15.775879 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet a063083f54ad49d2a842fa080d44ed85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a063083f54ad49d2a842fa080d44ed85 already in progress: creating tablet
I20251214 20:52:15.775966 1207 raft_consensus.cc:2272] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.776708 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 63ef97aeeaa94f4da683fbd133b05bd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b1a937150ce4ecfa210c8465d682496) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.776902 742 tablet_service.cc:1552] Processing DeleteTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.777160 1208 tablet_replica.cc:333] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:15.777383 1208 raft_consensus.cc:2243] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.777570 1208 raft_consensus.cc:2272] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.777097 739 tablet_service.cc:1505] Processing CreateTablet for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (DEFAULT_TABLE table=default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.778359 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719d157ac8f74eb7911df05ab5e4a3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.779409 1207 ts_tablet_manager.cc:1916] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:15.783201 523 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06381efe75284e649fd6d1a9903cfa9b consensus state has no leader
I20251214 20:52:15.783963 1526 raft_consensus.cc:493] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:15.784101 1526 raft_consensus.cc:515] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.784159 1208 ts_tablet_manager.cc:1916] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.784735 1526 leader_election.cc:290] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:15.785342 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12b5f4326aa0413eb303c132ba21decd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:15.785519 772 raft_consensus.cc:2468] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:15.785905 989 leader_election.cc:304] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
W20251214 20:52:15.786952 991 leader_election.cc:343] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 12b5f4326aa0413eb303c132ba21decd
I20251214 20:52:15.787272 1526 raft_consensus.cc:2804] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:15.787365 1526 raft_consensus.cc:493] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:15.787438 1526 raft_consensus.cc:3060] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:15.795535 1208 ts_tablet_manager.cc:1929] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.795775 1206 tablet_bootstrap.cc:492] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:15.795835 1208 log.cc:1199] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/63ef97aeeaa94f4da683fbd133b05bd1
I20251214 20:52:15.795969 1206 ts_tablet_manager.cc:1403] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.005s
I20251214 20:52:15.796233 1208 ts_tablet_manager.cc:1950] T 63ef97aeeaa94f4da683fbd133b05bd1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:15.796751 1206 raft_consensus.cc:359] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.796952 1206 raft_consensus.cc:385] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.797019 1206 raft_consensus.cc:740] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.797256 1206 consensus_queue.cc:260] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.797417 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 63ef97aeeaa94f4da683fbd133b05bd1 (table default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]) successfully deleted
I20251214 20:52:15.798281 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 251) took 1358 ms. Trace:
I20251214 20:52:15.798511 1205 rpcz_store.cc:276] 1214 20:52:14.439262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.446248 (+ 6986us) service_pool.cc:225] Handling call
1214 20:52:15.798041 (+1351793us) ts_tablet_manager.cc:1151] Deleting tablet 4b86f4adaf4540f48a4f42bc45e259e5
1214 20:52:15.798056 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.798211 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1351515}
I20251214 20:52:15.799340 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 248) took 1386 ms. Trace:
I20251214 20:52:15.799535 1526 raft_consensus.cc:515] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.801347 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 8d74d3abc45a41969289c0a0892bfa04 from {username='slave'} at 127.0.0.1:48876
W20251214 20:52:15.802557 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 4b86f4adaf4540f48a4f42bc45e259e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b86f4adaf4540f48a4f42bc45e259e5 already in progress: creating tablet
W20251214 20:52:15.804590 991 leader_election.cc:343] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 12b5f4326aa0413eb303c132ba21decd
I20251214 20:52:15.804872 892 tablet_service.cc:1505] Processing CreateTablet for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (DEFAULT_TABLE table=default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.805127 1206 ts_tablet_manager.cc:1434] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251214 20:52:15.805352 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719d157ac8f74eb7911df05ab5e4a3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.805850 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "12b5f4326aa0413eb303c132ba21decd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:15.806041 772 raft_consensus.cc:3060] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:15.808645 772 raft_consensus.cc:2468] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:15.809751 884 tablet_service.cc:1505] Processing CreateTablet for tablet 9e1835374cfa48939d72476870accc3b (DEFAULT_TABLE table=default.Table_dd97baf658ea4f44bba79f081b5ff75e [id=ba545f2c5ba949709f18c486d993faab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.810024 989 leader_election.cc:304] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:15.810210 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e1835374cfa48939d72476870accc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.811157 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 9e1835374cfa48939d72476870accc3b (DEFAULT_TABLE table=default.Table_dd97baf658ea4f44bba79f081b5ff75e [id=ba545f2c5ba949709f18c486d993faab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.811563 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e1835374cfa48939d72476870accc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.812659 1206 tablet_bootstrap.cc:492] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:15.814323 1206 tablet_bootstrap.cc:654] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.815198 1526 leader_election.cc:290] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:15.815290 1588 raft_consensus.cc:2804] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
W20251214 20:52:15.815424 526 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c973e0111804b8b869068bdcc5869bf consensus state has no leader
I20251214 20:52:15.816418 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (DEFAULT_TABLE table=default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.817421 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719d157ac8f74eb7911df05ab5e4a3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.822734 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 4d6e7306254044449f1127fb963ca5f0 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:15.799578 1205 rpcz_store.cc:276] 1214 20:52:14.412512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.472662 (+ 60150us) service_pool.cc:225] Handling call
1214 20:52:15.799145 (+1326483us) ts_tablet_manager.cc:1151] Deleting tablet c4b22043d81b493fb12116422b1d3fb1
1214 20:52:15.799154 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.799299 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326058}
W20251214 20:52:15.831689 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c4b22043d81b493fb12116422b1d3fb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4b22043d81b493fb12116422b1d3fb1 already in progress: creating tablet
I20251214 20:52:15.832803 1207 ts_tablet_manager.cc:1929] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.841621 1207 log.cc:1199] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/b0e805cc899347a083d4d549ad68e2f9
I20251214 20:52:15.842631 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_3f7396ac07d649fa8d342ae05765da80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:15.843405 521 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f7396ac07d649fa8d342ae05765da80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.845952 1207 ts_tablet_manager.cc:1950] T b0e805cc899347a083d4d549ad68e2f9 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:15.845957 1205 tablet_replica.cc:333] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:15.842439 1206 tablet_bootstrap.cc:492] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:15.848093 1206 ts_tablet_manager.cc:1403] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20251214 20:52:15.849262 884 tablet_service.cc:1505] Processing CreateTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 (DEFAULT_TABLE table=default.Table_893229542bec44e8b903bb0934bd16f5 [id=83b21f4baa574636ae3b50358fefae95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.849223 1206 raft_consensus.cc:359] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.849505 1206 raft_consensus.cc:385] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.849619 1206 raft_consensus.cc:740] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.849856 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f4ea72c6274569a98cab6afdc55d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.848165 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 221) took 1803 ms. Trace:
I20251214 20:52:15.849961 1207 rpcz_store.cc:276] 1214 20:52:14.044750 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.059245 (+ 14495us) service_pool.cc:225] Handling call
1214 20:52:15.775533 (+1716288us) ts_tablet_manager.cc:1151] Deleting tablet b0e805cc899347a083d4d549ad68e2f9
1214 20:52:15.775541 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.782334 (+ 6793us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.832676 (+ 50342us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.848100 (+ 15424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1703481}
I20251214 20:52:15.850255 1206 consensus_queue.cc:260] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.851257 1206 ts_tablet_manager.cc:1434] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:15.851408 1205 raft_consensus.cc:2243] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.851714 1205 raft_consensus.cc:2272] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.851974 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet b0e805cc899347a083d4d549ad68e2f9 (table default.Table_2695eb1d7e3a40c08c5a19eb6afce20f [id=ae6d8ad2d21d4208a1c19435e108f95b]) successfully deleted
I20251214 20:52:15.854094 1202 tablet_bootstrap.cc:492] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:15.854128 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 232) took 1476 ms. Trace:
I20251214 20:52:15.854302 1202 ts_tablet_manager.cc:1403] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.139s user 0.010s sys 0.035s
I20251214 20:52:15.854274 1207 rpcz_store.cc:276] 1214 20:52:14.377463 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.377926 (+ 463us) service_pool.cc:225] Handling call
1214 20:52:15.850249 (+1472323us) ts_tablet_manager.cc:1151] Deleting tablet 24ff58f321124d12b5330dca185d426e
1214 20:52:15.850259 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.854067 (+ 3808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472094}
W20251214 20:52:15.854952 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 24ff58f321124d12b5330dca185d426e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24ff58f321124d12b5330dca185d426e already in progress: creating tablet
I20251214 20:52:15.855221 1202 raft_consensus.cc:359] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.855475 1202 raft_consensus.cc:385] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.855576 1202 raft_consensus.cc:740] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.855927 1202 consensus_queue.cc:260] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.859226 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 231) took 1483 ms. Trace:
I20251214 20:52:15.859278 1205 ts_tablet_manager.cc:1916] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.859861 1207 rpcz_store.cc:276] 1214 20:52:14.376014 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.376088 (+ 74us) service_pool.cc:225] Handling call
1214 20:52:15.855313 (+1479225us) ts_tablet_manager.cc:1151] Deleting tablet 324379e3217442c3addcfe58060b9ad7
1214 20:52:15.858991 (+ 3678us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.859174 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1475836}
W20251214 20:52:15.860409 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 324379e3217442c3addcfe58060b9ad7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 324379e3217442c3addcfe58060b9ad7 already in progress: creating tablet
I20251214 20:52:15.859777 1206 tablet_bootstrap.cc:492] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:15.861042 1202 ts_tablet_manager.cc:1434] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:15.861433 1202 tablet_bootstrap.cc:492] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:15.862867 1206 tablet_bootstrap.cc:654] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.864344 1207 tablet_replica.cc:333] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:15.864323 1202 tablet_bootstrap.cc:654] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.867259 1035 tablet_service.cc:1505] Processing CreateTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 (DEFAULT_TABLE table=default.Table_893229542bec44e8b903bb0934bd16f5 [id=83b21f4baa574636ae3b50358fefae95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.867980 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f4ea72c6274569a98cab6afdc55d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.875730 1207 raft_consensus.cc:2243] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:15.872731 742 tablet_service.cc:1505] Processing CreateTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 (DEFAULT_TABLE table=default.Table_893229542bec44e8b903bb0934bd16f5 [id=83b21f4baa574636ae3b50358fefae95]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.873195 884 tablet_service.cc:1505] Processing CreateTablet for tablet c6a8863ace4541aea7463af24db27142 (DEFAULT_TABLE table=default.Table_88881a27865d4b0aa6db6a2b736f0c17 [id=e0d1675617c74aa8a8afd81bd90f9639]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.877903 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a8863ace4541aea7463af24db27142. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.876256 1207 raft_consensus.cc:2272] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.876719 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f4ea72c6274569a98cab6afdc55d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.881846 754 tablet_service.cc:1505] Processing CreateTablet for tablet 9e1835374cfa48939d72476870accc3b (DEFAULT_TABLE table=default.Table_dd97baf658ea4f44bba79f081b5ff75e [id=ba545f2c5ba949709f18c486d993faab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.886142 1588 raft_consensus.cc:697] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:15.886746 1588 consensus_queue.cc:237] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.889029 1207 ts_tablet_manager.cc:1916] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.897668 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e1835374cfa48939d72476870accc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.899291 1035 tablet_service.cc:1505] Processing CreateTablet for tablet c6a8863ace4541aea7463af24db27142 (DEFAULT_TABLE table=default.Table_88881a27865d4b0aa6db6a2b736f0c17 [id=e0d1675617c74aa8a8afd81bd90f9639]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.899832 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a8863ace4541aea7463af24db27142. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.900053 739 tablet_service.cc:1505] Processing CreateTablet for tablet c6a8863ace4541aea7463af24db27142 (DEFAULT_TABLE table=default.Table_88881a27865d4b0aa6db6a2b736f0c17 [id=e0d1675617c74aa8a8afd81bd90f9639]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:15.900424 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a8863ace4541aea7463af24db27142. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:15.904043 1205 ts_tablet_manager.cc:1929] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:15.904382 1205 log.cc:1199] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/dc442726ec0c456290ecf2892d0df957
I20251214 20:52:15.905306 1203 tablet_bootstrap.cc:492] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:15.905577 1203 ts_tablet_manager.cc:1403] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.184s user 0.005s sys 0.002s
I20251214 20:52:15.906442 1203 raft_consensus.cc:359] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.906643 1203 raft_consensus.cc:385] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.906703 1203 raft_consensus.cc:740] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.906939 1203 consensus_queue.cc:260] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.907735 1203 ts_tablet_manager.cc:1434] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:15.908071 1203 tablet_bootstrap.cc:492] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:15.909847 1203 tablet_bootstrap.cc:654] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.911638 1205 ts_tablet_manager.cc:1950] T dc442726ec0c456290ecf2892d0df957 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:15.913516 1203 tablet_bootstrap.cc:492] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:15.913703 1203 ts_tablet_manager.cc:1403] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20251214 20:52:15.914391 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 252) took 1464 ms. Trace:
I20251214 20:52:15.914744 1205 rpcz_store.cc:276] 1214 20:52:14.448715 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.449876 (+ 1161us) service_pool.cc:225] Handling call
1214 20:52:15.832169 (+1382293us) ts_tablet_manager.cc:1151] Deleting tablet dc442726ec0c456290ecf2892d0df957
1214 20:52:15.832184 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.888381 (+ 56197us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.898037 (+ 9656us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.913320 (+ 15283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1358967}
I20251214 20:52:15.915308 1587 raft_consensus.cc:493] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:15.915480 1587 raft_consensus.cc:515] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.915998 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet dc442726ec0c456290ecf2892d0df957 (table default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]) successfully deleted
I20251214 20:52:15.916139 1587 leader_election.cc:290] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:15.916531 1205 tablet_replica.cc:333] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:15.916740 1205 raft_consensus.cc:2243] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.916937 1205 raft_consensus.cc:2272] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:15.917099 712 leader_election.cc:343] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 84a0237e4ee34f94821d8647a400575b
I20251214 20:52:15.917512 1205 ts_tablet_manager.cc:1916] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.917699 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84a0237e4ee34f94821d8647a400575b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:15.914538 1203 raft_consensus.cc:359] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.918469 1203 raft_consensus.cc:385] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.918784 1203 raft_consensus.cc:740] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:15.918865 713 leader_election.cc:343] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:15.918998 713 leader_election.cc:304] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:15.920982 1202 tablet_bootstrap.cc:492] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:15.921185 1202 ts_tablet_manager.cc:1403] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.006s
I20251214 20:52:15.922195 1202 raft_consensus.cc:359] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.922402 1202 raft_consensus.cc:385] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.922492 1202 raft_consensus.cc:740] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.923306 1587 raft_consensus.cc:2749] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:15.924907 1207 ts_tablet_manager.cc:1929] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:15.925028 1207 log.cc:1199] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/3fd0fa0dad974c98bf6f52c8cfe83035
I20251214 20:52:15.919404 1203 consensus_queue.cc:260] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.925503 518 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]). Expected version 3 got 0
I20251214 20:52:15.925948 1203 ts_tablet_manager.cc:1434] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251214 20:52:15.926249 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_c3fc0fa4ec254ff889443791279bb8af" } new_table_name: "default.Table_b4334a1d99a54c2eabe2fb909563313c" modify_external_catalogs: true
I20251214 20:52:15.926375 1203 tablet_bootstrap.cc:492] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:15.927670 1207 ts_tablet_manager.cc:1950] T 3fd0fa0dad974c98bf6f52c8cfe83035 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:15.929759 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 235) took 1551 ms. Trace:
I20251214 20:52:15.929881 1207 rpcz_store.cc:276] 1214 20:52:14.377781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.378542 (+ 761us) service_pool.cc:225] Handling call
1214 20:52:15.864253 (+1485711us) ts_tablet_manager.cc:1151] Deleting tablet 3fd0fa0dad974c98bf6f52c8cfe83035
1214 20:52:15.864263 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.909107 (+ 44844us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.924756 (+ 15649us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.929725 (+ 4969us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1482165}
I20251214 20:52:15.930004 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48876:
name: "default.Table_26a82e34bb024bd89d6819c80056ba3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:15.930300 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 3fd0fa0dad974c98bf6f52c8cfe83035 (table default.Table_0173f37106d54db18ef3975bc6b0d185 [id=5c14810dd5364dcaa218b80c3756eadc]) successfully deleted
W20251214 20:52:15.930511 517 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26a82e34bb024bd89d6819c80056ba3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:15.931066 1202 consensus_queue.cc:260] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.931900 1202 ts_tablet_manager.cc:1434] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20251214 20:52:15.932581 1206 tablet_bootstrap.cc:492] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:15.932139 892 tablet_service.cc:1552] Processing DeleteTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.932948 1206 ts_tablet_manager.cc:1403] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.003s
I20251214 20:52:15.934521 1202 tablet_bootstrap.cc:492] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:15.935169 1208 tablet_replica.cc:333] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:15.935397 1208 raft_consensus.cc:2243] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.935604 1208 raft_consensus.cc:2272] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.936829 1202 tablet_bootstrap.cc:654] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.937794 1208 ts_tablet_manager.cc:1916] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.934088 1206 raft_consensus.cc:359] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.940469 1206 raft_consensus.cc:385] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:15.940563 1206 raft_consensus.cc:740] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:15.940889 1206 consensus_queue.cc:260] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:15.941449 1206 ts_tablet_manager.cc:1434] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251214 20:52:15.942173 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 242) took 1533 ms. Trace:
I20251214 20:52:15.942327 1206 tablet_bootstrap.cc:492] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:15.942973 1207 rpcz_store.cc:276] 1214 20:52:14.399495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.399545 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:15.932795 (+1533250us) ts_tablet_manager.cc:1151] Deleting tablet 4b86f4adaf4540f48a4f42bc45e259e5
1214 20:52:15.932804 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.933388 (+ 584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1532980}
W20251214 20:52:15.943794 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 4b86f4adaf4540f48a4f42bc45e259e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b86f4adaf4540f48a4f42bc45e259e5 already in progress: creating tablet
I20251214 20:52:15.949486 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 239) took 1565 ms. Trace:
I20251214 20:52:15.951675 1207 rpcz_store.cc:276] 1214 20:52:14.397027 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.398328 (+ 1301us) service_pool.cc:225] Handling call
1214 20:52:15.943335 (+1545007us) ts_tablet_manager.cc:1151] Deleting tablet c4b22043d81b493fb12116422b1d3fb1
1214 20:52:15.949223 (+ 5888us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.949432 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529195}
I20251214 20:52:15.953431 524 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet f84e9dfc23d34eb0a379e8ac5a442e59
I20251214 20:52:15.953500 1207 tablet_replica.cc:333] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
W20251214 20:52:15.953917 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c4b22043d81b493fb12116422b1d3fb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4b22043d81b493fb12116422b1d3fb1 already in progress: creating tablet
I20251214 20:52:15.954522 1207 raft_consensus.cc:2243] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.954718 1207 raft_consensus.cc:2272] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.955955 892 tablet_service.cc:1552] Processing DeleteTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.956086 1207 ts_tablet_manager.cc:1916] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.961797 1513 raft_consensus.cc:493] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:15.961984 1513 raft_consensus.cc:515] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:15.962275 742 tablet_service.cc:1552] Processing DeleteTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37374
I20251214 20:52:15.962390 1208 ts_tablet_manager.cc:1929] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.962566 1208 log.cc:1199] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/d768c45bc02b4d108798efbf3f6aaf7d
I20251214 20:52:15.962726 1513 leader_election.cc:290] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:15.963047 1208 ts_tablet_manager.cc:1950] T d768c45bc02b4d108798efbf3f6aaf7d P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:15.964895 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "649f5bd67cbb4f61b7066269f0bf9266" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:15.965405 848 leader_election.cc:343] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:15.966539 1206 tablet_bootstrap.cc:654] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.968508 1203 tablet_bootstrap.cc:654] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:15.969568 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet d768c45bc02b4d108798efbf3f6aaf7d (table default.Table_ca875b1006554ba6845c0cf23671a8da [id=8c192689a28f4314bb5608c08ea21a2e]) successfully deleted
W20251214 20:52:15.969867 512 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet f84e9dfc23d34eb0a379e8ac5a442e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f84e9dfc23d34eb0a379e8ac5a442e59 already in progress: creating tablet
I20251214 20:52:15.970897 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:60078
I20251214 20:52:15.971699 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "649f5bd67cbb4f61b7066269f0bf9266" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:15.976876 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:48876:
table { table_name: "default.Table_616b5b16d08b43eb8a04a21a352031fe" } new_table_name: "default.Table_b7bd23d0e40346ca98a98ca28dced554" modify_external_catalogs: true
I20251214 20:52:15.978674 525 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]). Expected version 3 got 0
I20251214 20:52:15.979278 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 9b1a937150ce4ecfa210c8465d682496 from {username='slave'} at 127.0.0.1:48876
W20251214 20:52:15.980444 851 leader_election.cc:343] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:15.980597 851 leader_election.cc:304] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:15.981055 1513 raft_consensus.cc:2749] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:15.983825 1207 ts_tablet_manager.cc:1929] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:15.984004 1207 log.cc:1199] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ed1d007f1ca24b7291262708ddfca3c3
I20251214 20:52:15.984054 892 tablet_service.cc:1552] Processing DeleteTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.984362 1207 ts_tablet_manager.cc:1950] T ed1d007f1ca24b7291262708ddfca3c3 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:15.984344 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 649f5bd67cbb4f61b7066269f0bf9266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:43856
I20251214 20:52:15.985199 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 247) took 1302 ms. Trace:
I20251214 20:52:15.988749 1208 tablet_replica.cc:333] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
W20251214 20:52:15.988982 512 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet d768c45bc02b4d108798efbf3f6aaf7d because the tablet was not found. No further retry: Not found: Tablet not found: d768c45bc02b4d108798efbf3f6aaf7d
I20251214 20:52:15.989048 1208 raft_consensus.cc:2243] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:15.989238 1208 raft_consensus.cc:2272] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.990636 1208 ts_tablet_manager.cc:1916] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:15.993604 1207 rpcz_store.cc:276] 1214 20:52:14.682175 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.683050 (+ 875us) service_pool.cc:225] Handling call
1214 20:52:15.952016 (+1268966us) ts_tablet_manager.cc:1151] Deleting tablet ed1d007f1ca24b7291262708ddfca3c3
1214 20:52:15.952915 (+ 899us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.973345 (+ 20430us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.983678 (+ 10333us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.985124 (+ 1446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076012}
I20251214 20:52:15.994526 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ed1d007f1ca24b7291262708ddfca3c3 (table default.Table_e3603e6d524c40dea2a1a88b79d63a8e [id=7febb36ade704d57bdfd4211054277ab]) successfully deleted
I20251214 20:52:15.995007 1207 tablet_replica.cc:333] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:15.995515 1207 raft_consensus.cc:2243] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:15.996191 1207 raft_consensus.cc:2272] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:15.997576 1207 ts_tablet_manager.cc:1916] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.000170 1206 tablet_bootstrap.cc:492] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.001574 1206 ts_tablet_manager.cc:1403] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.007s
I20251214 20:52:16.002581 1206 raft_consensus.cc:359] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.002844 1206 raft_consensus.cc:385] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.003028 1206 raft_consensus.cc:740] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.003353 1206 consensus_queue.cc:260] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.005762 1205 ts_tablet_manager.cc:1929] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:16.008143 1208 ts_tablet_manager.cc:1929] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:16.008332 1208 log.cc:1199] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:16.008476 1205 log.cc:1199] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/118dd44151a34ba3ae58f9fe1d3790d4
I20251214 20:52:16.008725 1206 ts_tablet_manager.cc:1434] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251214 20:52:16.008761 1208 ts_tablet_manager.cc:1950] T 649f5bd67cbb4f61b7066269f0bf9266 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:16.008888 1205 ts_tablet_manager.cc:1950] T 118dd44151a34ba3ae58f9fe1d3790d4 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:16.009729 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 253) took 1559 ms. Trace:
I20251214 20:52:16.010025 512 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 649f5bd67cbb4f61b7066269f0bf9266 (table default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]) successfully deleted
I20251214 20:52:16.011328 1205 rpcz_store.cc:276] 1214 20:52:14.450183 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.450245 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:15.916431 (+1466186us) ts_tablet_manager.cc:1151] Deleting tablet 118dd44151a34ba3ae58f9fe1d3790d4
1214 20:52:15.916442 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:15.977613 (+ 61171us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.981357 (+ 3744us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.009692 (+ 28335us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1431292}
I20251214 20:52:16.012408 1202 tablet_bootstrap.cc:492] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.012593 1202 ts_tablet_manager.cc:1403] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.014s
I20251214 20:52:16.012903 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 118dd44151a34ba3ae58f9fe1d3790d4 (table default.Table_e8acf2f630434ce3b7f06a4a01e4a2a3 [id=ea94fa24050345f98516132203c9704c]) successfully deleted
I20251214 20:52:16.013003 1206 tablet_bootstrap.cc:492] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.013330 1205 tablet_replica.cc:333] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:16.013494 1202 raft_consensus.cc:359] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.013690 1202 raft_consensus.cc:385] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.013758 1202 raft_consensus.cc:740] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.014027 1202 consensus_queue.cc:260] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.014469 1205 raft_consensus.cc:2243] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:16.014506 1202 ts_tablet_manager.cc:1434] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:16.018333 1202 tablet_bootstrap.cc:492] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.018584 1205 raft_consensus.cc:2272] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.020290 1206 tablet_bootstrap.cc:654] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.021584 1202 tablet_bootstrap.cc:654] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.023931 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu pid 493
I20251214 20:52:16.024670 1205 ts_tablet_manager.cc:1916] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.025765 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20251214 20:52:16.030113 1203 tablet_bootstrap.cc:492] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.030323 1203 ts_tablet_manager.cc:1403] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.104s user 0.004s sys 0.012s
I20251214 20:52:16.031351 1203 raft_consensus.cc:359] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.031599 1203 raft_consensus.cc:385] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.031675 1203 raft_consensus.cc:740] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.031965 1203 consensus_queue.cc:260] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.034147 1203 ts_tablet_manager.cc:1434] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251214 20:52:16.034492 1203 tablet_bootstrap.cc:492] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.036405 1203 tablet_bootstrap.cc:654] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.038041 1206 tablet_bootstrap.cc:492] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.038235 1206 ts_tablet_manager.cc:1403] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20251214 20:52:16.039160 1206 raft_consensus.cc:359] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.039459 1206 raft_consensus.cc:385] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.039568 1206 raft_consensus.cc:740] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.039865 1206 consensus_queue.cc:260] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.043083 1206 ts_tablet_manager.cc:1434] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251214 20:52:16.045646 1206 tablet_bootstrap.cc:492] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.047680 1205 ts_tablet_manager.cc:1929] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:16.046129 1207 ts_tablet_manager.cc:1929] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:16.048295 1207 log.cc:1199] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/dc442726ec0c456290ecf2892d0df957
I20251214 20:52:16.050093 1206 tablet_bootstrap.cc:654] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.050288 1205 log.cc:1199] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/376e5efede87410eb9f95f36bedebf9d
I20251214 20:52:16.050802 1205 ts_tablet_manager.cc:1950] T 376e5efede87410eb9f95f36bedebf9d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:16.050990 1202 tablet_bootstrap.cc:492] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.051204 1202 ts_tablet_manager.cc:1403] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.005s
I20251214 20:52:16.052026 1202 raft_consensus.cc:359] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.052223 1202 raft_consensus.cc:385] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.052316 1202 raft_consensus.cc:740] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.052554 1202 consensus_queue.cc:260] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.053112 1202 ts_tablet_manager.cc:1434] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:16.053422 1202 tablet_bootstrap.cc:492] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.052134 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 257) took 1380 ms. Trace:
I20251214 20:52:16.057667 1202 tablet_bootstrap.cc:654] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.053612 1205 rpcz_store.cc:276] 1214 20:52:14.672024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.672144 (+ 120us) service_pool.cc:225] Handling call
1214 20:52:16.013256 (+1341112us) ts_tablet_manager.cc:1151] Deleting tablet 376e5efede87410eb9f95f36bedebf9d
1214 20:52:16.013269 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.028822 (+ 15553us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.047516 (+ 18694us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.052067 (+ 4551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1340821}
I20251214 20:52:16.058982 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 269) took 1312 ms. Trace:
I20251214 20:52:16.059384 1205 rpcz_store.cc:276] 1214 20:52:14.746851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.746966 (+ 115us) service_pool.cc:225] Handling call
1214 20:52:16.058680 (+1311714us) ts_tablet_manager.cc:1151] Deleting tablet 8faa7fd7315f485c82aee38bcf4cd8fc
1214 20:52:16.058693 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.058875 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1311350}
I20251214 20:52:16.059989 1203 tablet_bootstrap.cc:492] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.060159 1203 ts_tablet_manager.cc:1403] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20251214 20:52:16.060271 1207 ts_tablet_manager.cc:1950] T dc442726ec0c456290ecf2892d0df957 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:16.061203 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 246) took 1379 ms. Trace:
I20251214 20:52:16.061110 1203 raft_consensus.cc:359] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.061328 1203 raft_consensus.cc:385] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.061426 1203 raft_consensus.cc:740] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.061316 1207 rpcz_store.cc:276] 1214 20:52:14.682041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.686570 (+ 4529us) service_pool.cc:225] Handling call
1214 20:52:15.994914 (+1308344us) ts_tablet_manager.cc:1151] Deleting tablet dc442726ec0c456290ecf2892d0df957
1214 20:52:15.994926 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.019895 (+ 24969us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.045954 (+ 26059us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.061163 (+ 15209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1111181}
I20251214 20:52:16.061663 1203 consensus_queue.cc:260] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.061761 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 262) took 1136 ms. Trace:
I20251214 20:52:16.061890 1207 rpcz_store.cc:276] 1214 20:52:14.925157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.925357 (+ 200us) service_pool.cc:225] Handling call
1214 20:52:16.061593 (+1136236us) ts_tablet_manager.cc:1151] Deleting tablet ec85857f047e44d78835305ddcb70768
1214 20:52:16.061602 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.061704 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1135967}
I20251214 20:52:16.062247 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 267) took 1136 ms. Trace:
I20251214 20:52:16.062363 1203 ts_tablet_manager.cc:1434] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:16.062347 1207 rpcz_store.cc:276] 1214 20:52:14.925703 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.925832 (+ 129us) service_pool.cc:225] Handling call
1214 20:52:16.062104 (+1136272us) ts_tablet_manager.cc:1151] Deleting tablet 35a529702f924a5c9bf53c3c668c6094
1214 20:52:16.062112 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.062209 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136115}
I20251214 20:52:16.062716 1203 tablet_bootstrap.cc:492] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.062702 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 259) took 1137 ms. Trace:
I20251214 20:52:16.062815 1207 rpcz_store.cc:276] 1214 20:52:14.924820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.926157 (+ 1337us) service_pool.cc:225] Handling call
1214 20:52:16.062560 (+1136403us) ts_tablet_manager.cc:1151] Deleting tablet 8faa7fd7315f485c82aee38bcf4cd8fc
1214 20:52:16.062568 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.062666 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136127}
I20251214 20:52:16.068059 1202 tablet_bootstrap.cc:492] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.068249 1202 ts_tablet_manager.cc:1403] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20251214 20:52:16.068645 1203 tablet_bootstrap.cc:654] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.070026 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 271) took 1312 ms. Trace:
I20251214 20:52:16.070518 1202 raft_consensus.cc:359] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.070719 1202 raft_consensus.cc:385] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.070793 1202 raft_consensus.cc:740] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.071081 1202 consensus_queue.cc:260] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.071363 1206 tablet_bootstrap.cc:492] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.071599 1206 ts_tablet_manager.cc:1403] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.011s
I20251214 20:52:16.072659 1202 ts_tablet_manager.cc:1434] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251214 20:52:16.072556 1206 raft_consensus.cc:359] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.072937 1206 raft_consensus.cc:385] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.073058 1206 raft_consensus.cc:740] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.073174 1202 tablet_bootstrap.cc:492] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.073307 1206 consensus_queue.cc:260] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.073824 1206 ts_tablet_manager.cc:1434] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:16.074306 1206 tablet_bootstrap.cc:492] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.074874 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 260) took 1149 ms. Trace:
I20251214 20:52:16.075058 1207 rpcz_store.cc:276] 1214 20:52:14.924943 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.926501 (+ 1558us) service_pool.cc:225] Handling call
1214 20:52:16.063010 (+1136509us) ts_tablet_manager.cc:1151] Deleting tablet a10663bd306d40e7a6375d06d69be22d
1214 20:52:16.063018 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.074806 (+ 11788us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136292}
I20251214 20:52:16.071084 1205 rpcz_store.cc:276] 1214 20:52:14.747089 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.747178 (+ 89us) service_pool.cc:225] Handling call
1214 20:52:16.059787 (+1312609us) ts_tablet_manager.cc:1151] Deleting tablet 9c0f49e5d78240a296e47eccb3218e4f
1214 20:52:16.059795 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.059901 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1312300}
I20251214 20:52:16.076056 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 270) took 1329 ms. Trace:
I20251214 20:52:16.076174 1205 rpcz_store.cc:276] 1214 20:52:14.746989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.757542 (+ 10553us) service_pool.cc:225] Handling call
1214 20:52:16.075881 (+1318339us) ts_tablet_manager.cc:1151] Deleting tablet a10663bd306d40e7a6375d06d69be22d
1214 20:52:16.075891 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.076006 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1317931}
I20251214 20:52:16.076333 1206 tablet_bootstrap.cc:654] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.076606 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 272) took 1314 ms. Trace:
I20251214 20:52:16.076722 1205 rpcz_store.cc:276] 1214 20:52:14.762422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.763185 (+ 763us) service_pool.cc:225] Handling call
1214 20:52:16.076477 (+1313292us) ts_tablet_manager.cc:1151] Deleting tablet ec85857f047e44d78835305ddcb70768
1214 20:52:16.076485 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.076561 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1313002}
I20251214 20:52:16.077291 1202 tablet_bootstrap.cc:654] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.080514 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 274) took 1298 ms. Trace:
I20251214 20:52:16.080665 1205 rpcz_store.cc:276] 1214 20:52:14.781116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.799095 (+ 17979us) service_pool.cc:225] Handling call
1214 20:52:16.077022 (+1277927us) ts_tablet_manager.cc:1151] Deleting tablet 5da9deeedce0421c9aeaebd71ba11b1b
1214 20:52:16.077030 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.079651 (+ 2621us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275779}
I20251214 20:52:16.081164 1205 tablet_replica.cc:333] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:16.081409 1205 raft_consensus.cc:2243] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:16.081645 1205 raft_consensus.cc:2272] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.084857 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 261) took 1159 ms. Trace:
I20251214 20:52:16.085052 1207 rpcz_store.cc:276] 1214 20:52:14.925040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.926759 (+ 1719us) service_pool.cc:225] Handling call
1214 20:52:16.075618 (+1148859us) ts_tablet_manager.cc:1151] Deleting tablet 9c0f49e5d78240a296e47eccb3218e4f
1214 20:52:16.077865 (+ 2247us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.084802 (+ 6937us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148610}
I20251214 20:52:16.085302 1203 tablet_bootstrap.cc:492] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.085457 1203 ts_tablet_manager.cc:1403] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20251214 20:52:16.085574 1207 tablet_replica.cc:333] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:16.085860 1207 raft_consensus.cc:2243] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:16.086061 1207 raft_consensus.cc:2272] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.086230 1203 raft_consensus.cc:359] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.086449 1203 raft_consensus.cc:385] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.086519 1203 raft_consensus.cc:740] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.086774 1205 ts_tablet_manager.cc:1916] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.086772 1203 consensus_queue.cc:260] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.088608 1203 ts_tablet_manager.cc:1434] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:16.088959 1203 tablet_bootstrap.cc:492] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.095263 1205 ts_tablet_manager.cc:1929] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:16.095961 1203 tablet_bootstrap.cc:654] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.096114 1205 log.cc:1199] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/63ef97aeeaa94f4da683fbd133b05bd1
I20251214 20:52:16.096668 1205 ts_tablet_manager.cc:1950] T 63ef97aeeaa94f4da683fbd133b05bd1 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:16.097065 1206 tablet_bootstrap.cc:492] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.097005 1587 raft_consensus.cc:493] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.097293 1587 raft_consensus.cc:515] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.097565 1207 ts_tablet_manager.cc:1916] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.097896 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 273) took 1317 ms. Trace:
I20251214 20:52:16.098053 1587 leader_election.cc:290] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.098026 1205 rpcz_store.cc:276] 1214 20:52:14.780326 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.805789 (+ 25463us) service_pool.cc:225] Handling call
1214 20:52:16.081074 (+1275285us) ts_tablet_manager.cc:1151] Deleting tablet 63ef97aeeaa94f4da683fbd133b05bd1
1214 20:52:16.081083 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.090645 (+ 9562us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.095068 (+ 4423us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.097857 (+ 2789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237226}
I20251214 20:52:16.098613 1205 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60078 (request call id 277) took 1258 ms. Trace:
I20251214 20:52:16.098713 1205 rpcz_store.cc:276] 1214 20:52:14.839678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.839761 (+ 83us) service_pool.cc:225] Handling call
1214 20:52:16.098447 (+1258686us) ts_tablet_manager.cc:1151] Deleting tablet 35a529702f924a5c9bf53c3c668c6094
1214 20:52:16.098457 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.098578 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1239492}
I20251214 20:52:16.098968 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "144dee1a6d7344f2b79da6e79e67599e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:16.099956 1205 tablet_replica.cc:333] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:16.100201 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "144dee1a6d7344f2b79da6e79e67599e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.100422 1056 raft_consensus.cc:2468] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:16.100657 1205 raft_consensus.cc:2243] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:16.100880 1205 raft_consensus.cc:2272] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.100841 915 raft_consensus.cc:2468] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:16.101395 712 leader_election.cc:304] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.101550 1205 ts_tablet_manager.cc:1916] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.102164 1587 raft_consensus.cc:2804] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.102273 1587 raft_consensus.cc:493] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.102373 1587 raft_consensus.cc:3060] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.106596 1587 raft_consensus.cc:515] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.107725 1205 ts_tablet_manager.cc:1929] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:16.107784 1202 tablet_bootstrap.cc:492] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.107789 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "144dee1a6d7344f2b79da6e79e67599e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:16.107987 1205 log.cc:1199] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/810ba69dda7f4631b1432bd99029609b
I20251214 20:52:16.108039 1202 ts_tablet_manager.cc:1403] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.004s
I20251214 20:52:16.108119 915 raft_consensus.cc:3060] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.108390 1205 ts_tablet_manager.cc:1950] T 810ba69dda7f4631b1432bd99029609b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:16.109071 1202 raft_consensus.cc:359] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.109300 1202 raft_consensus.cc:385] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.109400 1202 raft_consensus.cc:740] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.109414 1205 tablet_replica.cc:333] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:16.109613 1205 raft_consensus.cc:2243] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:16.109700 1202 consensus_queue.cc:260] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.109881 1205 pending_rounds.cc:70] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:16.110380 1202 ts_tablet_manager.cc:1434] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:16.109936 1205 pending_rounds.cc:77] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493714997506048 op_type: NO_OP noop_request { }
I20251214 20:52:16.110656 1205 raft_consensus.cc:2889] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:16.111102 1202 tablet_bootstrap.cc:492] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.111802 915 raft_consensus.cc:2468] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:16.112473 712 leader_election.cc:304] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.110793 1205 raft_consensus.cc:2272] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.112803 1465 raft_consensus.cc:2804] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.112833 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "144dee1a6d7344f2b79da6e79e67599e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:16.112987 1465 raft_consensus.cc:697] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:16.113035 1056 raft_consensus.cc:3060] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.113227 1465 consensus_queue.cc:237] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.113981 1205 ts_tablet_manager.cc:1916] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.114670 1207 ts_tablet_manager.cc:1929] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:16.114817 1207 log.cc:1199] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/63ef97aeeaa94f4da683fbd133b05bd1
I20251214 20:52:16.115198 1207 ts_tablet_manager.cc:1950] T 63ef97aeeaa94f4da683fbd133b05bd1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:16.115836 1056 raft_consensus.cc:2468] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:16.116614 1587 leader_election.cc:290] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.116858 1206 ts_tablet_manager.cc:1403] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.013s
I20251214 20:52:16.120342 1202 tablet_bootstrap.cc:654] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.121243 1206 raft_consensus.cc:359] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.121579 1206 raft_consensus.cc:385] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.121706 1206 raft_consensus.cc:740] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.122210 1206 consensus_queue.cc:260] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.123557 1206 ts_tablet_manager.cc:1434] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:16.126538 1203 tablet_bootstrap.cc:492] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.127633 1203 ts_tablet_manager.cc:1403] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.008s
I20251214 20:52:16.128273 1206 tablet_bootstrap.cc:492] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.127979 1205 ts_tablet_manager.cc:1929] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:16.129350 1205 log.cc:1199] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/12b5f4326aa0413eb303c132ba21decd
I20251214 20:52:16.130606 1205 ts_tablet_manager.cc:1950] T 12b5f4326aa0413eb303c132ba21decd P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:16.130571 1203 raft_consensus.cc:359] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.131224 1203 raft_consensus.cc:385] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.131321 1203 raft_consensus.cc:740] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.131556 1203 consensus_queue.cc:260] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.132284 1203 ts_tablet_manager.cc:1434] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:16.131053 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 264) took 1205 ms. Trace:
I20251214 20:52:16.132552 1207 rpcz_store.cc:276] 1214 20:52:14.925361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.925685 (+ 324us) service_pool.cc:225] Handling call
1214 20:52:16.085492 (+1159807us) ts_tablet_manager.cc:1151] Deleting tablet 63ef97aeeaa94f4da683fbd133b05bd1
1214 20:52:16.085504 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.100703 (+ 15199us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.114541 (+ 13838us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.130976 (+ 16435us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148261}
I20251214 20:52:16.133401 1206 tablet_bootstrap.cc:654] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.133431 1203 tablet_bootstrap.cc:492] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.134968 1205 tablet_replica.cc:333] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:16.135497 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 265) took 1210 ms. Trace:
I20251214 20:52:16.136013 1205 raft_consensus.cc:2243] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:16.136101 1202 tablet_bootstrap.cc:492] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.136269 1202 ts_tablet_manager.cc:1403] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.003s
I20251214 20:52:16.136327 1205 pending_rounds.cc:70] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:16.136406 1205 pending_rounds.cc:77] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493714427412480 op_type: NO_OP noop_request { }
I20251214 20:52:16.136554 1205 raft_consensus.cc:2889] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:16.136647 1205 raft_consensus.cc:2272] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.137060 1202 raft_consensus.cc:359] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.137252 1202 raft_consensus.cc:385] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.137321 1202 raft_consensus.cc:740] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.137564 1202 consensus_queue.cc:260] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.138072 1202 ts_tablet_manager.cc:1434] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:16.138336 1205 ts_tablet_manager.cc:1916] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.138691 1207 rpcz_store.cc:276] 1214 20:52:14.925449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.931104 (+ 5655us) service_pool.cc:225] Handling call
1214 20:52:16.135302 (+1204198us) ts_tablet_manager.cc:1151] Deleting tablet 5da9deeedce0421c9aeaebd71ba11b1b
1214 20:52:16.135313 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.135445 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195149}
I20251214 20:52:16.139204 1203 tablet_bootstrap.cc:654] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.139474 1207 tablet_replica.cc:333] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:16.143466 1202 tablet_bootstrap.cc:492] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.145665 1202 tablet_bootstrap.cc:654] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.145583 1207 raft_consensus.cc:2243] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:16.150991 1207 raft_consensus.cc:2272] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.151984 1205 ts_tablet_manager.cc:1929] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:16.152066 1207 ts_tablet_manager.cc:1916] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.152915 1203 tablet_bootstrap.cc:492] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.153226 1203 ts_tablet_manager.cc:1403] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20251214 20:52:16.154211 1203 raft_consensus.cc:359] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.154577 1203 raft_consensus.cc:385] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.154685 1203 raft_consensus.cc:740] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.155005 1203 consensus_queue.cc:260] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.156572 1203 ts_tablet_manager.cc:1434] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:16.156958 1203 tablet_bootstrap.cc:492] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.159066 1203 tablet_bootstrap.cc:654] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.161501 1202 tablet_bootstrap.cc:492] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.161682 1202 ts_tablet_manager.cc:1403] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20251214 20:52:16.161512 1205 log.cc:1199] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/b0e805cc899347a083d4d549ad68e2f9
I20251214 20:52:16.162582 1205 ts_tablet_manager.cc:1950] T b0e805cc899347a083d4d549ad68e2f9 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:16.162550 1465 raft_consensus.cc:493] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.162599 1202 raft_consensus.cc:359] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.162775 1202 raft_consensus.cc:385] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.162747 1465 raft_consensus.cc:515] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.162858 1202 raft_consensus.cc:740] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.163106 1202 consensus_queue.cc:260] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.163542 1465 leader_election.cc:290] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.163640 1202 ts_tablet_manager.cc:1434] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:16.164471 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b4f582d8165439d86d9228c146dc41a" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:16.165484 713 leader_election.cc:343] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.165928 1205 tablet_replica.cc:333] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:16.166148 1205 raft_consensus.cc:2243] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:16.166342 1205 raft_consensus.cc:2272] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.167055 1205 ts_tablet_manager.cc:1916] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.171494 1202 tablet_bootstrap.cc:492] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:16.171819 712 leader_election.cc:343] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 6b4f582d8165439d86d9228c146dc41a
I20251214 20:52:16.172014 712 leader_election.cc:304] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.172556 1465 raft_consensus.cc:2749] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:16.173727 1202 tablet_bootstrap.cc:654] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.173813 1203 tablet_bootstrap.cc:492] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.173972 1203 ts_tablet_manager.cc:1403] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20251214 20:52:16.174912 1465 raft_consensus.cc:493] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.174896 1203 raft_consensus.cc:359] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.175086 1203 raft_consensus.cc:385] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.175069 1465 raft_consensus.cc:515] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.175204 1203 raft_consensus.cc:740] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.175432 1203 consensus_queue.cc:260] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.175732 1207 ts_tablet_manager.cc:1929] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:16.175872 1207 log.cc:1199] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/376e5efede87410eb9f95f36bedebf9d
I20251214 20:52:16.178002 1203 ts_tablet_manager.cc:1434] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:16.178195 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13f93bb5e5f4b59bcf9f4bfed070f25" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:16.178364 1203 tablet_bootstrap.cc:492] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.178426 915 raft_consensus.cc:2468] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:16.179232 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13f93bb5e5f4b59bcf9f4bfed070f25" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.179486 1056 raft_consensus.cc:2468] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:16.180344 1588 raft_consensus.cc:493] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.180449 1203 tablet_bootstrap.cc:654] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.180553 712 leader_election.cc:304] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.180570 1588 raft_consensus.cc:515] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.180929 1587 raft_consensus.cc:2804] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.181035 1587 raft_consensus.cc:493] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.181116 1587 raft_consensus.cc:3060] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.181274 1588 leader_election.cc:290] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.182039 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84a0237e4ee34f94821d8647a400575b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.182271 772 raft_consensus.cc:2468] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:16.182750 1205 ts_tablet_manager.cc:1929] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:16.183682 989 leader_election.cc:304] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:16.183800 1205 log.cc:1199] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/8e2ee5cc95304fc1a80ed4d31838ac38
I20251214 20:52:16.184177 1205 ts_tablet_manager.cc:1950] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:16.184608 991 leader_election.cc:343] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 84a0237e4ee34f94821d8647a400575b
I20251214 20:52:16.184864 1587 raft_consensus.cc:515] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.186631 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13f93bb5e5f4b59bcf9f4bfed070f25" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:16.186890 915 raft_consensus.cc:3060] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.189394 1588 raft_consensus.cc:2804] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.189489 1588 raft_consensus.cc:493] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.189574 1588 raft_consensus.cc:3060] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.189805 915 raft_consensus.cc:2468] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:16.190591 712 leader_election.cc:304] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.190989 1476 raft_consensus.cc:2804] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.175874 1465 leader_election.cc:290] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.191946 1202 tablet_bootstrap.cc:492] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.192111 1202 ts_tablet_manager.cc:1403] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20251214 20:52:16.192423 1588 raft_consensus.cc:515] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.192621 1203 tablet_bootstrap.cc:492] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.192803 1203 ts_tablet_manager.cc:1403] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20251214 20:52:16.192894 1202 raft_consensus.cc:359] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.193071 1202 raft_consensus.cc:385] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.193137 1202 raft_consensus.cc:740] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.193341 1202 consensus_queue.cc:260] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.193706 1203 raft_consensus.cc:359] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.193924 1203 raft_consensus.cc:385] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.193997 1203 raft_consensus.cc:740] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.194044 1202 ts_tablet_manager.cc:1434] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:16.194339 1202 tablet_bootstrap.cc:492] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.194294 1203 consensus_queue.cc:260] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.194917 1203 ts_tablet_manager.cc:1434] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:16.195369 1203 tablet_bootstrap.cc:492] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.195634 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d13f93bb5e5f4b59bcf9f4bfed070f25" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:16.195840 1056 raft_consensus.cc:3060] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.196590 1202 tablet_bootstrap.cc:654] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.197229 1203 tablet_bootstrap.cc:654] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:16.197819 991 leader_election.cc:343] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 84a0237e4ee34f94821d8647a400575b
I20251214 20:52:16.198987 1056 raft_consensus.cc:2468] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:16.200045 1587 leader_election.cc:290] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.200418 1203 tablet_bootstrap.cc:492] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.200575 1203 ts_tablet_manager.cc:1403] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251214 20:52:16.202796 1202 tablet_bootstrap.cc:492] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.202967 1202 ts_tablet_manager.cc:1403] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251214 20:52:16.203747 1202 raft_consensus.cc:359] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.203938 1202 raft_consensus.cc:385] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.204033 1202 raft_consensus.cc:740] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.202040 1203 raft_consensus.cc:359] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.204310 1203 raft_consensus.cc:385] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.204272 1202 consensus_queue.cc:260] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.204429 1203 raft_consensus.cc:740] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.204660 1203 consensus_queue.cc:260] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.204749 1202 ts_tablet_manager.cc:1434] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:16.205382 1203 ts_tablet_manager.cc:1434] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:16.206149 1202 tablet_bootstrap.cc:492] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.206812 1203 tablet_bootstrap.cc:492] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.206954 1587 raft_consensus.cc:493] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.207273 1587 raft_consensus.cc:515] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.208038 1202 tablet_bootstrap.cc:654] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:16.208887 712 leader_election.cc:343] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 569a02142c71427d91868b02695ae12b
I20251214 20:52:16.209869 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "569a02142c71427d91868b02695ae12b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.210150 1056 raft_consensus.cc:2468] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:16.210584 713 leader_election.cc:304] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.210927 1465 raft_consensus.cc:2804] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.211040 1465 raft_consensus.cc:493] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.211117 1465 raft_consensus.cc:3060] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.211325 1203 tablet_bootstrap.cc:654] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.212491 1587 leader_election.cc:290] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.213876 1588 leader_election.cc:290] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.214231 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84a0237e4ee34f94821d8647a400575b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:16.214452 772 raft_consensus.cc:3060] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.214429 1465 raft_consensus.cc:515] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.216967 1207 ts_tablet_manager.cc:1950] T 376e5efede87410eb9f95f36bedebf9d P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:16.217767 1206 tablet_bootstrap.cc:492] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.219894 1203 tablet_bootstrap.cc:492] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.220070 1203 ts_tablet_manager.cc:1403] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20251214 20:52:16.220257 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "569a02142c71427d91868b02695ae12b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:16.220517 1056 raft_consensus.cc:3060] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.220968 1203 raft_consensus.cc:359] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.221179 1203 raft_consensus.cc:385] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.221251 1203 raft_consensus.cc:740] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.221572 1203 consensus_queue.cc:260] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.223740 1203 ts_tablet_manager.cc:1434] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:16.223922 1056 raft_consensus.cc:2468] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:16.224074 1203 tablet_bootstrap.cc:492] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.217856 772 raft_consensus.cc:2468] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:16.218565 1465 leader_election.cc:290] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:16.219732 712 leader_election.cc:343] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 569a02142c71427d91868b02695ae12b
I20251214 20:52:16.225706 713 leader_election.cc:304] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.226140 989 leader_election.cc:304] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.226356 1465 raft_consensus.cc:2804] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.226548 1465 raft_consensus.cc:697] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:16.226629 1588 raft_consensus.cc:2804] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.226781 1588 raft_consensus.cc:697] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:16.227070 1588 consensus_queue.cc:237] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.227209 1465 consensus_queue.cc:237] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.228353 1206 ts_tablet_manager.cc:1403] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.100s user 0.028s sys 0.006s
I20251214 20:52:16.217859 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 268) took 1277 ms. Trace:
I20251214 20:52:16.229354 1206 raft_consensus.cc:359] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.229415 1207 rpcz_store.cc:276] 1214 20:52:14.940847 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.941440 (+ 593us) service_pool.cc:225] Handling call
1214 20:52:16.139381 (+1197941us) ts_tablet_manager.cc:1151] Deleting tablet 376e5efede87410eb9f95f36bedebf9d
1214 20:52:16.139391 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.164732 (+ 25341us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.175578 (+ 10846us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.217809 (+ 42231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197756}
I20251214 20:52:16.230357 1207 tablet_replica.cc:333] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:16.224843 1476 raft_consensus.cc:697] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:16.231015 1207 raft_consensus.cc:2243] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:16.231416 1207 raft_consensus.cc:2272] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.231340 1476 consensus_queue.cc:237] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.232247 1207 ts_tablet_manager.cc:1916] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.233282 1513 raft_consensus.cc:493] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.233450 1513 raft_consensus.cc:515] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.234136 1513 leader_election.cc:290] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.234783 1203 tablet_bootstrap.cc:654] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.235293 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2842f1dc0c374d24946f90dac1cf8aa5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:16.235839 848 leader_election.cc:343] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.236233 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2842f1dc0c374d24946f90dac1cf8aa5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.236791 1202 tablet_bootstrap.cc:492] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
W20251214 20:52:16.236797 851 leader_election.cc:343] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.236925 851 leader_election.cc:304] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:16.236980 1202 ts_tablet_manager.cc:1403] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20251214 20:52:16.237362 1513 raft_consensus.cc:2749] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:16.237741 1206 raft_consensus.cc:385] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.237849 1202 raft_consensus.cc:359] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.238066 1202 raft_consensus.cc:385] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.238157 1202 raft_consensus.cc:740] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.238376 1206 raft_consensus.cc:740] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.238405 1202 consensus_queue.cc:260] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.239411 1206 consensus_queue.cc:260] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.240538 1206 ts_tablet_manager.cc:1434] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20251214 20:52:16.240887 1202 ts_tablet_manager.cc:1434] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:16.242069 1206 tablet_bootstrap.cc:492] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.243330 1202 tablet_bootstrap.cc:492] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.245095 1203 tablet_bootstrap.cc:492] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.245270 1203 ts_tablet_manager.cc:1403] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20251214 20:52:16.246112 1203 raft_consensus.cc:359] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.246311 1203 raft_consensus.cc:385] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.246397 1203 raft_consensus.cc:740] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.246656 1203 consensus_queue.cc:260] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.249753 1202 tablet_bootstrap.cc:654] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.250250 1203 ts_tablet_manager.cc:1434] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20251214 20:52:16.250609 1203 tablet_bootstrap.cc:492] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.256006 1203 tablet_bootstrap.cc:654] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.264394 1203 tablet_bootstrap.cc:492] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.264611 1203 ts_tablet_manager.cc:1403] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20251214 20:52:16.265563 1203 raft_consensus.cc:359] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.265780 1203 raft_consensus.cc:385] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.265851 1203 raft_consensus.cc:740] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.266175 1203 consensus_queue.cc:260] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.266404 1207 ts_tablet_manager.cc:1929] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:16.266944 1207 log.cc:1199] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/810ba69dda7f4631b1432bd99029609b
I20251214 20:52:16.267484 1207 ts_tablet_manager.cc:1950] T 810ba69dda7f4631b1432bd99029609b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:16.268421 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 269) took 1327 ms. Trace:
I20251214 20:52:16.268604 1207 rpcz_store.cc:276] 1214 20:52:14.940975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:14.965877 (+ 24902us) service_pool.cc:225] Handling call
1214 20:52:16.230262 (+1264385us) ts_tablet_manager.cc:1151] Deleting tablet 810ba69dda7f4631b1432bd99029609b
1214 20:52:16.230277 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.235747 (+ 5470us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.266262 (+ 30515us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:16.268367 (+ 2105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1238794}
I20251214 20:52:16.269182 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 281) took 1180 ms. Trace:
I20251214 20:52:16.269748 1203 ts_tablet_manager.cc:1434] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:16.270094 1203 tablet_bootstrap.cc:492] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.271301 1202 tablet_bootstrap.cc:492] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.271449 1202 ts_tablet_manager.cc:1403] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.002s
I20251214 20:52:16.271571 1207 rpcz_store.cc:276] 1214 20:52:15.089108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.092914 (+ 3806us) service_pool.cc:225] Handling call
1214 20:52:16.268998 (+1176084us) ts_tablet_manager.cc:1151] Deleting tablet 3bbb28259c1442ae94f4eff76c25c539
1214 20:52:16.269007 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.269135 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1174317}
I20251214 20:52:16.272229 1202 raft_consensus.cc:359] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.272405 1202 raft_consensus.cc:385] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.272490 1202 raft_consensus.cc:740] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.271095 1206 tablet_bootstrap.cc:654] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.272720 1202 consensus_queue.cc:260] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.274338 1202 ts_tablet_manager.cc:1434] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:16.275756 1202 tablet_bootstrap.cc:492] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.277602 1203 tablet_bootstrap.cc:654] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.280052 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 282) took 1190 ms. Trace:
I20251214 20:52:16.280618 1202 tablet_bootstrap.cc:654] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.280745 1207 rpcz_store.cc:276] 1214 20:52:15.089296 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.092952 (+ 3656us) service_pool.cc:225] Handling call
1214 20:52:16.279859 (+1186907us) ts_tablet_manager.cc:1151] Deleting tablet 8d9f74fb2c6244a6887e3c922262bbab
1214 20:52:16.279868 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.279999 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1183698}
I20251214 20:52:16.281380 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 283) took 1185 ms. Trace:
I20251214 20:52:16.283186 1207 rpcz_store.cc:276] 1214 20:52:15.095949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.096228 (+ 279us) service_pool.cc:225] Handling call
1214 20:52:16.281221 (+1184993us) ts_tablet_manager.cc:1151] Deleting tablet c2ae9dc8ead8403cb7215af8778872f0
1214 20:52:16.281230 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.281331 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1184793}
I20251214 20:52:16.283783 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 284) took 1184 ms. Trace:
I20251214 20:52:16.283922 1207 rpcz_store.cc:276] 1214 20:52:15.099659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.099917 (+ 258us) service_pool.cc:225] Handling call
1214 20:52:16.283642 (+1183725us) ts_tablet_manager.cc:1151] Deleting tablet bb698ff30fcc47d09000cc68d2c5225b
1214 20:52:16.283651 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.283735 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1181445}
I20251214 20:52:16.284572 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 289) took 1075 ms. Trace:
I20251214 20:52:16.284960 1207 rpcz_store.cc:276] 1214 20:52:15.208668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.209912 (+ 1244us) service_pool.cc:225] Handling call
1214 20:52:16.284445 (+1074533us) ts_tablet_manager.cc:1151] Deleting tablet f0d661a1bb034053a4634ff537eafc7d
1214 20:52:16.284453 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.284528 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1070912}
I20251214 20:52:16.285671 1207 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37374 (request call id 291) took 1055 ms. Trace:
I20251214 20:52:16.286062 1207 rpcz_store.cc:276] 1214 20:52:15.229827 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.233884 (+ 4057us) service_pool.cc:225] Handling call
1214 20:52:16.285533 (+1051649us) ts_tablet_manager.cc:1151] Deleting tablet b315744c8cd043c5a8bdceec73a7e28b
1214 20:52:16.285541 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:16.285625 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051291}
I20251214 20:52:16.292603 1202 tablet_bootstrap.cc:492] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.292809 1465 raft_consensus.cc:493] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.292874 1202 ts_tablet_manager.cc:1403] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20251214 20:52:16.293916 1206 tablet_bootstrap.cc:492] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.293845 1202 raft_consensus.cc:359] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.294118 1202 raft_consensus.cc:385] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.294205 1202 raft_consensus.cc:740] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.294284 1465 raft_consensus.cc:515] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.294500 1202 consensus_queue.cc:260] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.295004 1465 leader_election.cc:290] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.295469 1206 ts_tablet_manager.cc:1403] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.002s
I20251214 20:52:16.296291 1206 raft_consensus.cc:359] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.296550 1206 raft_consensus.cc:385] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.296662 1206 raft_consensus.cc:740] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.296844 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba50e32f7e724868bb891045a939e362" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:16.297005 1206 consensus_queue.cc:260] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.297088 915 raft_consensus.cc:2468] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:16.297515 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba50e32f7e724868bb891045a939e362" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.297719 1056 raft_consensus.cc:2468] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:16.297946 1206 ts_tablet_manager.cc:1434] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:16.298049 1207 tablet_replica.cc:333] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:16.300524 712 leader_election.cc:304] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.301223 1202 ts_tablet_manager.cc:1434] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20251214 20:52:16.301582 1465 raft_consensus.cc:2804] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.301672 1465 raft_consensus.cc:493] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.301687 1202 tablet_bootstrap.cc:492] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.301759 1465 raft_consensus.cc:3060] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.304590 1202 tablet_bootstrap.cc:654] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.305284 1206 tablet_bootstrap.cc:492] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.308481 1206 tablet_bootstrap.cc:654] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.309950 1465 raft_consensus.cc:515] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.310689 1203 tablet_bootstrap.cc:492] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.310889 1203 ts_tablet_manager.cc:1403] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.019s
I20251214 20:52:16.310875 1465 leader_election.cc:290] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.311208 1207 raft_consensus.cc:2243] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:16.311301 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba50e32f7e724868bb891045a939e362" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:16.311492 1207 pending_rounds.cc:70] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:16.311509 915 raft_consensus.cc:3060] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.311547 1207 pending_rounds.cc:77] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493713234489344 op_type: NO_OP noop_request { }
I20251214 20:52:16.311654 1207 raft_consensus.cc:2889] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:16.311736 1207 raft_consensus.cc:2272] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.311839 1203 raft_consensus.cc:359] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.312039 1203 raft_consensus.cc:385] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.312129 1203 raft_consensus.cc:740] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.312386 1203 consensus_queue.cc:260] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.313128 1207 ts_tablet_manager.cc:1916] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.314090 915 raft_consensus.cc:2468] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:16.314713 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba50e32f7e724868bb891045a939e362" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:16.314914 1056 raft_consensus.cc:3060] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.316062 1203 ts_tablet_manager.cc:1434] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251214 20:52:16.316514 1203 tablet_bootstrap.cc:492] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.317801 1056 raft_consensus.cc:2468] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:16.318653 712 leader_election.cc:304] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.318974 1476 raft_consensus.cc:493] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.319195 1203 tablet_bootstrap.cc:654] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.319193 1476 raft_consensus.cc:515] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.319969 1476 leader_election.cc:290] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.320140 1476 raft_consensus.cc:2804] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.320284 1476 raft_consensus.cc:697] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:16.320611 1476 consensus_queue.cc:237] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.322438 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8faa7fd7315f485c82aee38bcf4cd8fc" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:16.323269 712 leader_election.cc:343] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 8faa7fd7315f485c82aee38bcf4cd8fc
I20251214 20:52:16.324553 1056 raft_consensus.cc:2468] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:16.325129 713 leader_election.cc:304] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.325613 1625 raft_consensus.cc:2804] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.325706 1625 raft_consensus.cc:493] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.325809 1625 raft_consensus.cc:3060] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.327178 1202 tablet_bootstrap.cc:492] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.327385 1202 ts_tablet_manager.cc:1403] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.007s
I20251214 20:52:16.328418 1202 raft_consensus.cc:359] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.328527 1625 raft_consensus.cc:515] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.328639 1202 raft_consensus.cc:385] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.328729 1202 raft_consensus.cc:740] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.329068 1202 consensus_queue.cc:260] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.329152 1625 leader_election.cc:290] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.330695 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8faa7fd7315f485c82aee38bcf4cd8fc" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:16.330894 1054 raft_consensus.cc:3060] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.332800 1207 ts_tablet_manager.cc:1929] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:16.332921 1207 log.cc:1199] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/118dd44151a34ba3ae58f9fe1d3790d4
I20251214 20:52:16.333312 1207 ts_tablet_manager.cc:1950] T 118dd44151a34ba3ae58f9fe1d3790d4 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:16.333467 1054 raft_consensus.cc:2468] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:16.334144 713 leader_election.cc:304] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:16.334252 1207 tablet_replica.cc:333] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:16.334403 1465 raft_consensus.cc:493] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.334549 1465 raft_consensus.cc:515] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.334425 1207 raft_consensus.cc:2243] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:16.335074 1465 leader_election.cc:290] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.335253 1465 raft_consensus.cc:2804] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.335271 1202 ts_tablet_manager.cc:1434] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251214 20:52:16.335373 1465 raft_consensus.cc:697] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:16.335436 1207 raft_consensus.cc:2272] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.335635 1465 consensus_queue.cc:237] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.335944 1207 ts_tablet_manager.cc:1916] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.339202 1202 tablet_bootstrap.cc:492] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:16.339682 712 leader_election.cc:343] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 8e2ee5cc95304fc1a80ed4d31838ac38
W20251214 20:52:16.340298 712 leader_election.cc:343] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 8faa7fd7315f485c82aee38bcf4cd8fc
W20251214 20:52:16.341023 713 leader_election.cc:343] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 8e2ee5cc95304fc1a80ed4d31838ac38
I20251214 20:52:16.341142 713 leader_election.cc:304] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.341454 1202 tablet_bootstrap.cc:654] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.341588 1625 raft_consensus.cc:2749] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:16.350544 1207 ts_tablet_manager.cc:1929] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:16.353910 1207 log.cc:1199] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/12b5f4326aa0413eb303c132ba21decd
I20251214 20:52:16.354617 1207 ts_tablet_manager.cc:1950] T 12b5f4326aa0413eb303c132ba21decd P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:16.357450 1206 tablet_bootstrap.cc:492] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.357625 1206 ts_tablet_manager.cc:1403] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.003s
I20251214 20:52:16.357822 1202 tablet_bootstrap.cc:492] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.358069 1202 ts_tablet_manager.cc:1403] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20251214 20:52:16.359040 1202 raft_consensus.cc:359] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.359288 1202 raft_consensus.cc:385] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.359372 1202 raft_consensus.cc:740] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.359480 1206 raft_consensus.cc:359] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.359717 1206 raft_consensus.cc:385] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.359860 1206 raft_consensus.cc:740] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.360109 1207 tablet_replica.cc:333] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:16.363572 1202 consensus_queue.cc:260] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.360150 1206 consensus_queue.cc:260] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.364096 1207 raft_consensus.cc:2243] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:16.364298 1207 raft_consensus.cc:2272] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.364351 1206 ts_tablet_manager.cc:1434] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:16.364681 1206 tablet_bootstrap.cc:492] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.364900 1207 ts_tablet_manager.cc:1916] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.366990 1206 tablet_bootstrap.cc:654] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.369338 1202 ts_tablet_manager.cc:1434] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20251214 20:52:16.373330 1513 raft_consensus.cc:493] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.373517 1513 raft_consensus.cc:515] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.374126 1513 leader_election.cc:290] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.374835 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "472e4a5990e64d02b1d52dc6e86d7190" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:16.376128 848 leader_election.cc:343] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.377507 1202 tablet_bootstrap.cc:492] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.380131 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "472e4a5990e64d02b1d52dc6e86d7190" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:16.380738 851 leader_election.cc:343] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.380923 851 leader_election.cc:304] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:16.381175 1513 raft_consensus.cc:2749] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:16.381446 1206 tablet_bootstrap.cc:492] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.381671 1206 ts_tablet_manager.cc:1403] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20251214 20:52:16.381796 1202 tablet_bootstrap.cc:654] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.382514 1203 tablet_bootstrap.cc:492] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.382699 1203 ts_tablet_manager.cc:1403] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.066s user 0.022s sys 0.022s
I20251214 20:52:16.382658 1206 raft_consensus.cc:359] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.382990 1206 raft_consensus.cc:385] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.383102 1206 raft_consensus.cc:740] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.383477 1206 consensus_queue.cc:260] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.383648 1203 raft_consensus.cc:359] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.383829 1203 raft_consensus.cc:385] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.383896 1203 raft_consensus.cc:740] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.384275 1206 ts_tablet_manager.cc:1434] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:16.384781 1206 tablet_bootstrap.cc:492] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.386458 1206 tablet_bootstrap.cc:654] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.387966 1207 ts_tablet_manager.cc:1929] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:16.388211 1207 log.cc:1199] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/84a0237e4ee34f94821d8647a400575b
I20251214 20:52:16.388626 1207 ts_tablet_manager.cc:1950] T 84a0237e4ee34f94821d8647a400575b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:16.390934 1206 tablet_bootstrap.cc:492] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.391235 1206 ts_tablet_manager.cc:1403] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251214 20:52:16.392345 1206 raft_consensus.cc:359] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.392596 1206 raft_consensus.cc:385] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.392717 1206 raft_consensus.cc:740] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.392998 1206 consensus_queue.cc:260] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.393726 1206 ts_tablet_manager.cc:1434] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:16.395401 1203 consensus_queue.cc:260] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.396152 1203 ts_tablet_manager.cc:1434] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251214 20:52:16.397022 1203 tablet_bootstrap.cc:492] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.397719 1207 tablet_replica.cc:333] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:16.398069 1207 raft_consensus.cc:2243] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:16.398428 1207 pending_rounds.cc:70] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:16.398552 1207 pending_rounds.cc:77] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493716389486592 op_type: NO_OP noop_request { }
I20251214 20:52:16.398806 1207 raft_consensus.cc:2889] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:16.398859 1203 tablet_bootstrap.cc:654] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.399045 1207 raft_consensus.cc:2272] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.400086 1207 ts_tablet_manager.cc:1916] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.407505 1206 tablet_bootstrap.cc:492] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.413625 1513 raft_consensus.cc:493] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.413908 1513 raft_consensus.cc:515] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.414865 1513 leader_election.cc:290] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.415797 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fa0317e999b483788e5c5652e6e83de" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.416069 1056 raft_consensus.cc:2468] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:16.416157 1203 tablet_bootstrap.cc:492] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.416323 1203 ts_tablet_manager.cc:1403] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20251214 20:52:16.417124 851 leader_election.cc:304] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.417135 1203 raft_consensus.cc:359] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.417316 1203 raft_consensus.cc:385] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.417394 1203 raft_consensus.cc:740] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.417688 1203 consensus_queue.cc:260] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.418177 1203 ts_tablet_manager.cc:1434] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:16.418108 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fa0317e999b483788e5c5652e6e83de" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.418403 772 raft_consensus.cc:2468] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:16.418533 1203 tablet_bootstrap.cc:492] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.419530 1513 raft_consensus.cc:2804] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.419674 1513 raft_consensus.cc:493] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.419764 1513 raft_consensus.cc:3060] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.420272 1203 tablet_bootstrap.cc:654] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.425446 1513 raft_consensus.cc:515] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.426505 1207 ts_tablet_manager.cc:1929] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:16.426741 1207 log.cc:1199] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/569a02142c71427d91868b02695ae12b
I20251214 20:52:16.426929 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fa0317e999b483788e5c5652e6e83de" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:16.427277 772 raft_consensus.cc:3060] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.427335 1207 ts_tablet_manager.cc:1950] T 569a02142c71427d91868b02695ae12b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:16.428994 1207 tablet_replica.cc:333] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:16.429234 1207 raft_consensus.cc:2243] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:16.429430 1207 raft_consensus.cc:2272] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:16.430009 1207 ts_tablet_manager.cc:1916] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:16.430738 772 raft_consensus.cc:2468] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:16.431628 848 leader_election.cc:304] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.435868 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3fa0317e999b483788e5c5652e6e83de" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:16.436158 1056 raft_consensus.cc:3060] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.440177 1056 raft_consensus.cc:2468] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:16.441614 1513 leader_election.cc:290] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.442664 1513 raft_consensus.cc:2804] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.443105 1513 raft_consensus.cc:697] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:16.443831 1513 consensus_queue.cc:237] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.444232 1206 tablet_bootstrap.cc:654] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.447857 1207 ts_tablet_manager.cc:1929] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:16.448119 1207 log.cc:1199] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/8e2ee5cc95304fc1a80ed4d31838ac38
I20251214 20:52:16.448531 1207 ts_tablet_manager.cc:1950] T 8e2ee5cc95304fc1a80ed4d31838ac38 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:16.468106 1588 raft_consensus.cc:493] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.468418 1588 raft_consensus.cc:515] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.469384 1588 leader_election.cc:290] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.470494 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023299180d074893847e84dcd9812751" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:16.470779 915 raft_consensus.cc:2468] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:16.471909 991 leader_election.cc:304] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.473021 1588 raft_consensus.cc:2804] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.473174 1588 raft_consensus.cc:493] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.473260 1588 raft_consensus.cc:3060] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.476559 1588 raft_consensus.cc:515] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.478487 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023299180d074893847e84dcd9812751" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:16.478734 915 raft_consensus.cc:3060] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.482120 915 raft_consensus.cc:2468] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:16.482810 991 leader_election.cc:304] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.483237 1595 raft_consensus.cc:2804] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.485908 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023299180d074893847e84dcd9812751" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:16.487268 989 leader_election.cc:343] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.489171 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023299180d074893847e84dcd9812751" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:16.489718 989 leader_election.cc:343] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.490407 1588 leader_election.cc:290] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.495538 1655 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6713a20a82b434f8a8bc122113e26ad)
I20251214 20:52:16.495870 1655 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:16.499186 463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 423 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a" is_pre_election: true
I20251214 20:52:16.501457 1655 leader_election.cc:290] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819)
I20251214 20:52:16.502210 1656 raft_consensus.cc:493] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6713a20a82b434f8a8bc122113e26ad)
I20251214 20:52:16.502492 1656 raft_consensus.cc:515] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:16.504297 1656 leader_election.cc:290] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:16.526599 1650 raft_consensus.cc:493] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.532637 1334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 4 candidate_status { last_received { term: 3 index: 423 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" is_pre_election: true
I20251214 20:52:16.533169 1334 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c27224391721467a8f46e467a980b37a in term 3.
I20251214 20:52:16.533702 441 leader_election.cc:304] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, c27224391721467a8f46e467a980b37a; no voters:
I20251214 20:52:16.534129 1656 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251214 20:52:16.534271 1656 raft_consensus.cc:493] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 3 FOLLOWER]: Starting leader election (detected failure of leader e6713a20a82b434f8a8bc122113e26ad)
I20251214 20:52:16.534361 1656 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 3 FOLLOWER]: Advancing to term 4
I20251214 20:52:16.536130 1206 tablet_bootstrap.cc:492] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.536321 1206 ts_tablet_manager.cc:1403] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.129s user 0.000s sys 0.007s
I20251214 20:52:16.537267 1656 raft_consensus.cc:515] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:16.537426 1206 raft_consensus.cc:359] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.538638 1656 leader_election.cc:290] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 4 election: Requested vote from peers e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:16.538702 1206 raft_consensus.cc:385] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.539067 1650 raft_consensus.cc:515] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.539309 1334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 4 candidate_status { last_received { term: 3 index: 423 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4"
I20251214 20:52:16.539516 1334 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 3 FOLLOWER]: Advancing to term 4
I20251214 20:52:16.540964 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b652a8a80246443491551e76e68d2a8b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:16.541585 851 leader_election.cc:343] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.541909 1650 leader_election.cc:290] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.534689 1513 raft_consensus.cc:493] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.542145 1513 raft_consensus.cc:515] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.542548 1334 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c27224391721467a8f46e467a980b37a in term 4.
I20251214 20:52:16.543109 441 leader_election.cc:304] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, c27224391721467a8f46e467a980b37a; no voters:
I20251214 20:52:16.543519 1656 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 4 FOLLOWER]: Leader election won for term 4
I20251214 20:52:16.543637 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bd6856b800247de896f16eee66f2827" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.543866 1056 raft_consensus.cc:2468] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:16.544153 1513 leader_election.cc:290] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:16.534978 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 103387us
I20251214 20:52:16.544603 851 leader_election.cc:304] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.545825 1513 raft_consensus.cc:2804] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.545936 1513 raft_consensus.cc:493] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.546056 1513 raft_consensus.cc:3060] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.546545 1650 raft_consensus.cc:493] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.546766 1650 raft_consensus.cc:515] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.548074 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a142157acbcd44adb6099b4428103ee5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:16.548713 851 leader_election.cc:343] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.549131 1513 raft_consensus.cc:515] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.549808 1595 raft_consensus.cc:697] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:16.550522 1654 raft_consensus.cc:493] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.551007 1654 raft_consensus.cc:515] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.550748 1595 consensus_queue.cc:237] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
W20251214 20:52:16.554363 991 leader_election.cc:343] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:16.554484 1656 raft_consensus.cc:697] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 4 LEADER]: Becoming Leader. State: Replica: c27224391721467a8f46e467a980b37a, State: Running, Role: LEADER
I20251214 20:52:16.554908 1656 consensus_queue.cc:237] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 423, Committed index: 423, Last appended: 3.423, Last appended by leader: 423, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:16.555251 1654 leader_election.cc:290] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:16.539820 1476 raft_consensus.cc:493] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.555542 1476 raft_consensus.cc:515] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.556278 1476 leader_election.cc:290] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.545446 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b652a8a80246443491551e76e68d2a8b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.546075 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bd6856b800247de896f16eee66f2827" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.549372 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a142157acbcd44adb6099b4428103ee5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.552335 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "649f5bd67cbb4f61b7066269f0bf9266" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.557567 774 raft_consensus.cc:2468] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:16.557647 1659 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27224391721467a8f46e467a980b37a. Latest consensus state: current_term: 4 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:16.549630 1650 leader_election.cc:290] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.549701 1202 tablet_bootstrap.cc:492] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.558024 1202 ts_tablet_manager.cc:1403] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.181s user 0.091s sys 0.037s
I20251214 20:52:16.558302 989 leader_election.cc:304] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.559093 1659 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: LEADER
W20251214 20:52:16.559311 712 leader_election.cc:343] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: b1df0dad2fb040a5a12e0401e00c87db
I20251214 20:52:16.559298 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bd6856b800247de896f16eee66f2827" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:16.559532 1054 raft_consensus.cc:3060] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.559880 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1df0dad2fb040a5a12e0401e00c87db" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.560155 1056 raft_consensus.cc:2468] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:16.560631 713 leader_election.cc:304] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.561231 1476 raft_consensus.cc:2804] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.561350 1476 raft_consensus.cc:493] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.561434 1476 raft_consensus.cc:3060] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.562072 1054 raft_consensus.cc:2468] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:16.562839 1202 raft_consensus.cc:359] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.563050 1202 raft_consensus.cc:385] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.563146 1202 raft_consensus.cc:740] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.563407 1202 consensus_queue.cc:260] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.563925 1202 ts_tablet_manager.cc:1434] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251214 20:52:16.564239 1202 tablet_bootstrap.cc:492] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.565989 1202 tablet_bootstrap.cc:654] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.567821 851 leader_election.cc:304] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.569274 1650 raft_consensus.cc:2804] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.569413 1650 raft_consensus.cc:697] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:16.569685 1650 consensus_queue.cc:237] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.570326 1476 raft_consensus.cc:515] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:16.572196 712 leader_election.cc:343] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: b1df0dad2fb040a5a12e0401e00c87db
I20251214 20:52:16.576723 1588 raft_consensus.cc:2804] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.576893 1588 raft_consensus.cc:493] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.577001 1588 raft_consensus.cc:3060] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.578311 1513 leader_election.cc:290] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.580597 1588 raft_consensus.cc:515] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.582933 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "649f5bd67cbb4f61b7066269f0bf9266" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:16.583213 774 raft_consensus.cc:3060] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.586229 774 raft_consensus.cc:2468] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:16.587292 989 leader_election.cc:304] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:16.590991 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1df0dad2fb040a5a12e0401e00c87db" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:16.591287 1054 raft_consensus.cc:3060] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.591106 1476 leader_election.cc:290] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.593766 1054 raft_consensus.cc:2468] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
W20251214 20:52:16.595023 991 leader_election.cc:343] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:16.595690 713 leader_election.cc:304] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.596588 1476 raft_consensus.cc:2804] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.597357 1476 raft_consensus.cc:697] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:16.597792 1476 consensus_queue.cc:237] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.599648 1588 leader_election.cc:290] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:16.600111 1588 raft_consensus.cc:493] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.600706 1588 raft_consensus.cc:515] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.602180 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bd6856b800247de896f16eee66f2827" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:16.602445 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 376e5efede87410eb9f95f36bedebf9d (table default.Table_2dab958dcddb48e28bfc246163f539ae [id=0b01345b8b0b44c0b59bced8e8b89a75]) successfully deleted
I20251214 20:52:16.603494 535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 4 candidate_status { last_received { term: 3 index: 423 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
I20251214 20:52:16.605031 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0279992d1f2419ab1caae116a0391b2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:16.605312 915 raft_consensus.cc:2468] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:16.605854 991 leader_election.cc:304] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:16.603583 848 leader_election.cc:343] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.606979 848 leader_election.cc:304] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:16.607785 1588 leader_election.cc:290] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:16.608542 848 leader_election.cc:343] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.608042 1654 raft_consensus.cc:2804] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.610047 1654 raft_consensus.cc:697] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:16.610982 1588 raft_consensus.cc:2804] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251214 20:52:16.611265 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 06381efe75284e649fd6d1a9903cfa9b (table default.Table_7bc0f81d22a643589e8a54ceb2477807 [id=db33edd24660463ea648be7da745e29c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 06381efe75284e649fd6d1a9903cfa9b consensus state has no leader
W20251214 20:52:16.611599 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8faa7fd7315f485c82aee38bcf4cd8fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8faa7fd7315f485c82aee38bcf4cd8fc already in progress: creating tablet
W20251214 20:52:16.611891 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 9c0f49e5d78240a296e47eccb3218e4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c0f49e5d78240a296e47eccb3218e4f already in progress: creating tablet
W20251214 20:52:16.612038 848 leader_election.cc:343] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:16.612109 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a10663bd306d40e7a6375d06d69be22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a10663bd306d40e7a6375d06d69be22d already in progress: creating tablet
I20251214 20:52:16.612155 848 leader_election.cc:304] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:16.612282 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ec85857f047e44d78835305ddcb70768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec85857f047e44d78835305ddcb70768 already in progress: creating tablet
W20251214 20:52:16.612450 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 5da9deeedce0421c9aeaebd71ba11b1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5da9deeedce0421c9aeaebd71ba11b1b already in progress: creating tablet
I20251214 20:52:16.612437 1513 raft_consensus.cc:2749] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:16.612564 848 leader_election.cc:343] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.612617 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 63ef97aeeaa94f4da683fbd133b05bd1 (table default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]) successfully deleted
I20251214 20:52:16.608779 1653 raft_consensus.cc:2749] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:16.613186 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 35a529702f924a5c9bf53c3c668c6094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a529702f924a5c9bf53c3c668c6094 already in progress: creating tablet
W20251214 20:52:16.613375 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet bb698ff30fcc47d09000cc68d2c5225b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb698ff30fcc47d09000cc68d2c5225b already in progress: creating tablet
W20251214 20:52:16.613549 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c2ae9dc8ead8403cb7215af8778872f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2ae9dc8ead8403cb7215af8778872f0 already in progress: creating tablet
I20251214 20:52:16.613458 1588 raft_consensus.cc:493] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.613709 1588 raft_consensus.cc:3060] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:16.613718 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8d9f74fb2c6244a6887e3c922262bbab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9f74fb2c6244a6887e3c922262bbab already in progress: creating tablet
W20251214 20:52:16.613893 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3bbb28259c1442ae94f4eff76c25c539 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bbb28259c1442ae94f4eff76c25c539 already in progress: creating tablet
W20251214 20:52:16.614053 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f0d661a1bb034053a4634ff537eafc7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0d661a1bb034053a4634ff537eafc7d already in progress: creating tablet
W20251214 20:52:16.614248 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b315744c8cd043c5a8bdceec73a7e28b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b315744c8cd043c5a8bdceec73a7e28b already in progress: creating tablet
W20251214 20:52:16.614411 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 7fcfcfe81fe24d369eea696efc61d21c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fcfcfe81fe24d369eea696efc61d21c already in progress: creating tablet
I20251214 20:52:16.614565 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 810ba69dda7f4631b1432bd99029609b (table default.Table_850408f19d2848309b88410e331f0798 [id=c544a159297d49c8be1c9a0eff9986cf]) successfully deleted
I20251214 20:52:16.614704 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 12b5f4326aa0413eb303c132ba21decd (table default.Table_1437581b4e464656a47dc3ab99914c22 [id=b5ef4fb9aeba4c3fb423322b5f1d0e14]) successfully deleted
W20251214 20:52:16.614883 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2a7fc15882e2474bb6885070be13b1fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a7fc15882e2474bb6885070be13b1fa already in progress: creating tablet
I20251214 20:52:16.615037 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet b0e805cc899347a083d4d549ad68e2f9 (table default.Table_2695eb1d7e3a40c08c5a19eb6afce20f [id=ae6d8ad2d21d4208a1c19435e108f95b]) successfully deleted
W20251214 20:52:16.615235 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3d949abef1cb41a9905482d81a98d2c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d949abef1cb41a9905482d81a98d2c1 already in progress: creating tablet
W20251214 20:52:16.615398 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3e5e220d083243eb8aa4328f366cd7ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e5e220d083243eb8aa4328f366cd7ff already in progress: creating tablet
W20251214 20:52:16.615563 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 47e813373fbd482298312ead4f037c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e813373fbd482298312ead4f037c01 already in progress: creating tablet
I20251214 20:52:16.615556 1654 consensus_queue.cc:237] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.617409 535 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 4 candidate_status { last_received { term: 3 index: 423 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
I20251214 20:52:16.617861 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 4 candidate_status { last_received { term: 3 index: 423 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad"
I20251214 20:52:16.618610 1334 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 4 FOLLOWER]: Rejecting Update request from peer e6713a20a82b434f8a8bc122113e26ad for earlier term 3. Current term is 4. Ops: []
I20251214 20:52:16.619465 1176 consensus_queue.cc:1059] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 }, Status: INVALID_TERM, Last received: 3.423, Next index: 424, Last known committed idx: 423, Time since last communication: 0.000s
I20251214 20:52:16.620165 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet dc442726ec0c456290ecf2892d0df957 (table default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]) successfully deleted
W20251214 20:52:16.620398 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ec85857f047e44d78835305ddcb70768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec85857f047e44d78835305ddcb70768 already in progress: creating tablet
I20251214 20:52:16.620546 1206 raft_consensus.cc:740] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.621122 1206 consensus_queue.cc:260] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:16.621336 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet bf8643c7e4094f529cd55656c7b3edb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf8643c7e4094f529cd55656c7b3edb6 already in progress: creating tablet
I20251214 20:52:16.621536 513 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 8e2ee5cc95304fc1a80ed4d31838ac38 (table default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d]) successfully deleted
W20251214 20:52:16.621694 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 569a02142c71427d91868b02695ae12b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 569a02142c71427d91868b02695ae12b already in progress: creating tablet
W20251214 20:52:16.621853 513 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f84e9dfc23d34eb0a379e8ac5a442e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f84e9dfc23d34eb0a379e8ac5a442e59 already in progress: creating tablet
I20251214 20:52:16.622180 1206 ts_tablet_manager.cc:1434] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.086s user 0.004s sys 0.000s
I20251214 20:52:16.622227 1297 leader_election.cc:304] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4; no voters: c27224391721467a8f46e467a980b37a, e6713a20a82b434f8a8bc122113e26ad
I20251214 20:52:16.622587 1206 tablet_bootstrap.cc:492] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.622736 1655 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251214 20:52:16.624912 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 35a529702f924a5c9bf53c3c668c6094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a529702f924a5c9bf53c3c668c6094 already in progress: creating tablet
W20251214 20:52:16.625157 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8faa7fd7315f485c82aee38bcf4cd8fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8faa7fd7315f485c82aee38bcf4cd8fc already in progress: creating tablet
W20251214 20:52:16.625315 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet a10663bd306d40e7a6375d06d69be22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a10663bd306d40e7a6375d06d69be22d already in progress: creating tablet
W20251214 20:52:16.625478 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 9c0f49e5d78240a296e47eccb3218e4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c0f49e5d78240a296e47eccb3218e4f already in progress: creating tablet
I20251214 20:52:16.625602 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 63ef97aeeaa94f4da683fbd133b05bd1 (table default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]) successfully deleted
W20251214 20:52:16.625761 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 5da9deeedce0421c9aeaebd71ba11b1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5da9deeedce0421c9aeaebd71ba11b1b already in progress: creating tablet
I20251214 20:52:16.625891 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 376e5efede87410eb9f95f36bedebf9d (table default.Table_2dab958dcddb48e28bfc246163f539ae [id=0b01345b8b0b44c0b59bced8e8b89a75]) successfully deleted
I20251214 20:52:16.626034 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 810ba69dda7f4631b1432bd99029609b (table default.Table_850408f19d2848309b88410e331f0798 [id=c544a159297d49c8be1c9a0eff9986cf]) successfully deleted
W20251214 20:52:16.626173 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3bbb28259c1442ae94f4eff76c25c539 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bbb28259c1442ae94f4eff76c25c539 already in progress: creating tablet
W20251214 20:52:16.626331 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8d9f74fb2c6244a6887e3c922262bbab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9f74fb2c6244a6887e3c922262bbab already in progress: creating tablet
W20251214 20:52:16.626474 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c2ae9dc8ead8403cb7215af8778872f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2ae9dc8ead8403cb7215af8778872f0 already in progress: creating tablet
W20251214 20:52:16.626648 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet bb698ff30fcc47d09000cc68d2c5225b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb698ff30fcc47d09000cc68d2c5225b already in progress: creating tablet
W20251214 20:52:16.626787 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f0d661a1bb034053a4634ff537eafc7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0d661a1bb034053a4634ff537eafc7d already in progress: creating tablet
W20251214 20:52:16.626937 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b315744c8cd043c5a8bdceec73a7e28b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b315744c8cd043c5a8bdceec73a7e28b already in progress: creating tablet
I20251214 20:52:16.627074 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 118dd44151a34ba3ae58f9fe1d3790d4 (table default.Table_e8acf2f630434ce3b7f06a4a01e4a2a3 [id=ea94fa24050345f98516132203c9704c]) successfully deleted
I20251214 20:52:16.636114 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 12b5f4326aa0413eb303c132ba21decd (table default.Table_1437581b4e464656a47dc3ab99914c22 [id=b5ef4fb9aeba4c3fb423322b5f1d0e14]) successfully deleted
W20251214 20:52:16.636343 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 7fcfcfe81fe24d369eea696efc61d21c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fcfcfe81fe24d369eea696efc61d21c already in progress: creating tablet
I20251214 20:52:16.636504 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 84a0237e4ee34f94821d8647a400575b (table default.Table_2162a5b765254842bb0b7e4ba7f66136 [id=fa79ab98ae4f405d830ed263d3c0b3c1]) successfully deleted
W20251214 20:52:16.636646 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2a7fc15882e2474bb6885070be13b1fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a7fc15882e2474bb6885070be13b1fa already in progress: creating tablet
W20251214 20:52:16.636802 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3e5e220d083243eb8aa4328f366cd7ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e5e220d083243eb8aa4328f366cd7ff already in progress: creating tablet
W20251214 20:52:16.636945 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3d949abef1cb41a9905482d81a98d2c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d949abef1cb41a9905482d81a98d2c1 already in progress: creating tablet
W20251214 20:52:16.637126 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 47e813373fbd482298312ead4f037c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e813373fbd482298312ead4f037c01 already in progress: creating tablet
I20251214 20:52:16.637262 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 569a02142c71427d91868b02695ae12b (table default.Table_7b427a00182f4e059423bf0e18859b90 [id=17c89c03be10405f834504b803e594c7]) successfully deleted
I20251214 20:52:16.637405 510 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 8e2ee5cc95304fc1a80ed4d31838ac38 (table default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d]) successfully deleted
W20251214 20:52:16.637542 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet bf8643c7e4094f529cd55656c7b3edb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf8643c7e4094f529cd55656c7b3edb6 already in progress: creating tablet
W20251214 20:52:16.637702 510 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f84e9dfc23d34eb0a379e8ac5a442e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f84e9dfc23d34eb0a379e8ac5a442e59 already in progress: creating tablet
I20251214 20:52:16.638258 1203 tablet_bootstrap.cc:492] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.638500 1203 ts_tablet_manager.cc:1403] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.220s user 0.027s sys 0.044s
I20251214 20:52:16.638633 463 raft_consensus.cc:1240] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 4 LEADER]: Rejecting Update request from peer e6713a20a82b434f8a8bc122113e26ad for earlier term 3. Current term is 4. Ops: []
I20251214 20:52:16.640443 1203 raft_consensus.cc:359] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.640895 1203 raft_consensus.cc:385] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.640987 1203 raft_consensus.cc:740] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.641275 1203 consensus_queue.cc:260] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.644104 1176 consensus_queue.cc:1059] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 }, Status: INVALID_TERM, Last received: 3.423, Next index: 424, Last known committed idx: 423, Time since last communication: 0.000s
I20251214 20:52:16.644371 1176 consensus_queue.cc:1243] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [LEADER]: Peer c27224391721467a8f46e467a980b37a log is divergent from this leader: its last log entry 4.424 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 423
I20251214 20:52:16.656247 1202 tablet_bootstrap.cc:492] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.656464 1202 ts_tablet_manager.cc:1403] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.092s user 0.016s sys 0.017s
W20251214 20:52:16.656873 1173 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.012s sys 0.018s
W20251214 20:52:16.656968 1173 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.659s user 0.012s sys 0.018s
I20251214 20:52:16.657202 1173 raft_consensus.cc:3055] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 3 LEADER]: Stepping down as leader of term 3
I20251214 20:52:16.657280 1173 raft_consensus.cc:740] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e6713a20a82b434f8a8bc122113e26ad, State: Running, Role: LEADER
I20251214 20:52:16.657315 1202 raft_consensus.cc:359] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.657477 1202 raft_consensus.cc:385] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.657537 1202 raft_consensus.cc:740] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.657604 1173 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 423, Committed index: 423, Last appended: 3.423, Last appended by leader: 423, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:16.657747 1202 consensus_queue.cc:260] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.658113 1173 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 3 FOLLOWER]: Advancing to term 4
I20251214 20:52:16.658257 1202 ts_tablet_manager.cc:1434] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:16.658552 1202 tablet_bootstrap.cc:492] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:16.658543 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 71292.4us
I20251214 20:52:16.659380 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0279992d1f2419ab1caae116a0391b2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.660171 1202 tablet_bootstrap.cc:654] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:16.660749 989 leader_election.cc:343] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.662070 1202 tablet_bootstrap.cc:492] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.662213 1202 ts_tablet_manager.cc:1403] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251214 20:52:16.663007 1202 raft_consensus.cc:359] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.663205 1202 raft_consensus.cc:385] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.663305 1202 raft_consensus.cc:740] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.663543 1202 consensus_queue.cc:260] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.663651 520 master_service.cc:954] ReplaceTablet: received request to replace tablet b2e83bca8ed642b6a1b705ef70fde27f from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:16.664160 1202 ts_tablet_manager.cc:1434] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:16.664448 1202 tablet_bootstrap.cc:492] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.664863 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 3fd0fa0dad974c98bf6f52c8cfe83035 from {username='slave'} at 127.0.0.1:48876
I20251214 20:52:16.666647 1202 tablet_bootstrap.cc:654] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:16.670532 519 catalog_manager.cc:2202] Illegal state: an error occurred while writing to the sys-catalog: Replica e6713a20a82b434f8a8bc122113e26ad is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:16.670944 519 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48876 (request call id 448) took 1047 ms. Trace:
W20251214 20:52:16.671404 522 catalog_manager.cc:2202] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.670s ago.
I20251214 20:52:16.671639 522 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48876 (request call id 449) took 1031 ms. Trace:
W20251214 20:52:16.671888 523 catalog_manager.cc:6689] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.670s ago.
I20251214 20:52:16.672065 523 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:48876 (request call id 450) took 1028 ms. Trace:
W20251214 20:52:16.672283 526 catalog_manager.cc:6689] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.671s ago.
I20251214 20:52:16.672454 526 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:48876 (request call id 451) took 1011 ms. Trace:
W20251214 20:52:16.672663 521 catalog_manager.cc:2202] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.671s ago.
I20251214 20:52:16.672873 521 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:48876 (request call id 452) took 1001 ms. Trace:
W20251214 20:52:16.674546 518 catalog_manager.cc:3922] Illegal state: an error occurred while updating the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.672s ago.
I20251214 20:52:16.674760 1588 raft_consensus.cc:515] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.675588 1588 leader_election.cc:290] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.676434 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0279992d1f2419ab1caae116a0391b2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:16.680434 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0279992d1f2419ab1caae116a0391b2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:16.680709 915 raft_consensus.cc:3060] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.682315 1206 tablet_bootstrap.cc:654] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.683063 1588 raft_consensus.cc:493] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:16.683250 517 catalog_manager.cc:2202] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.677s ago.
I20251214 20:52:16.683292 1588 raft_consensus.cc:515] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:16.685876 524 catalog_manager.cc:3922] Illegal state: an error occurred while updating the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.685s ago.
W20251214 20:52:16.687557 525 catalog_manager.cc:6689] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.688s ago.
W20251214 20:52:16.687647 1114 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.61:34819)
W20251214 20:52:16.688045 1114 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.61:34819)
W20251214 20:52:16.689013 1114 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.61:34819)
W20251214 20:52:16.689327 1114 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.61:34819)
W20251214 20:52:16.691411 991 leader_election.cc:343] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: d768c45bc02b4d108798efbf3f6aaf7d
I20251214 20:52:16.691740 1588 leader_election.cc:290] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.693081 1206 tablet_bootstrap.cc:492] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.693243 1206 ts_tablet_manager.cc:1403] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.002s
I20251214 20:52:16.694124 1206 raft_consensus.cc:359] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.694324 1206 raft_consensus.cc:385] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.694413 1206 raft_consensus.cc:740] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.672520 526 rpcz_store.cc:276] 1214 20:52:15.660743 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.822657 (+161914us) service_pool.cc:225] Handling call
1214 20:52:16.672409 (+849752us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:16.669215 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:16.669302 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:16.669306 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:16.669309 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:16.669381 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:16.669389 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:16.669394 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:16.669447 (+ 53us) tablet.cc:699] Row locks acquired
1214 20:52:16.669449 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:16.669481 (+ 32us) op_driver.cc:389] HandleFailure(Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.671s ago.)
1214 20:52:16.669488 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:16.669496 (+ 8us) write_op.cc:454] Released schema lock
1214 20:52:16.669500 (+ 4us) write_op.cc:335] FINISH: Op aborted
1214 20:52:16.669555 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:16.669586 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":840271,"prepare.run_cpu_time_us":339,"prepare.run_wall_time_us":339}]]}
I20251214 20:52:16.672148 523 rpcz_store.cc:276] 1214 20:52:15.644001 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.801287 (+157286us) service_pool.cc:225] Handling call
1214 20:52:16.672038 (+870751us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:16.668736 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:16.668819 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:16.668823 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:16.668826 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:16.668909 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:16.668918 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:16.668923 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:16.668979 (+ 56us) tablet.cc:699] Row locks acquired
1214 20:52:16.668981 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:16.669014 (+ 33us) op_driver.cc:389] HandleFailure(Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.670s ago.)
1214 20:52:16.669034 (+ 20us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:16.669043 (+ 9us) write_op.cc:454] Released schema lock
1214 20:52:16.669047 (+ 4us) write_op.cc:335] FINISH: Op aborted
1214 20:52:16.669103 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:16.669134 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":866418,"prepare.run_cpu_time_us":365,"prepare.run_wall_time_us":365}]]}
I20251214 20:52:16.698791 915 raft_consensus.cc:2468] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:16.699472 991 leader_election.cc:304] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.672942 521 rpcz_store.cc:276] 1214 20:52:15.671040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.842440 (+171400us) service_pool.cc:225] Handling call
1214 20:52:15.843807 (+ 1367us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:15.844432 (+ 625us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:16.672827 (+828395us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:16.669674 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:16.669764 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:16.669768 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:16.669770 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:16.669829 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:16.669837 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:16.669842 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:16.669894 (+ 52us) tablet.cc:699] Row locks acquired
1214 20:52:16.669896 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:16.669936 (+ 40us) op_driver.cc:389] HandleFailure(Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.671s ago.)
1214 20:52:16.669942 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:16.669951 (+ 9us) write_op.cc:454] Released schema lock
1214 20:52:16.669955 (+ 4us) write_op.cc:335] FINISH: Op aborted
1214 20:52:16.670007 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:16.670042 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":824754,"prepare.run_cpu_time_us":333,"prepare.run_wall_time_us":333}]]}
I20251214 20:52:16.671067 519 rpcz_store.cc:276] 1214 20:52:15.623795 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.742429 (+118634us) service_pool.cc:225] Handling call
1214 20:52:15.745815 (+ 3386us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:15.746237 (+ 422us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:16.670911 (+924674us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:16.014068 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:16.015010 (+ 942us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:16.015015 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:16.015017 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:16.015159 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:16.015173 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:16.015178 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:16.015256 (+ 78us) tablet.cc:699] Row locks acquired
1214 20:52:16.015259 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:16.015300 (+ 41us) write_op.cc:270] Start()
1214 20:52:16.015335 (+ 35us) write_op.cc:276] Timestamp: P: 1765745536015262 usec, L: 0
1214 20:52:16.015339 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:16.667737 (+652398us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x616000041254. stack: 00007f0327b88ee6 00007f0327b88ca0 00007f0326b643ad 0000000000984f94 0000000000984f5d 000000000098492a 00007f0333a47a9b 00007f0334e2a875 00007f0334e29399 00007f0334e317c0 00007f0334e315cd 0000000000a8970c 00007f0327bcf20f 00007f0327bd252b 00007f0327bd23a8 0000000000a8970c
1214 20:52:16.667793 (+ 56us) op_driver.cc:389] HandleFailure(Illegal state: Replica e6713a20a82b434f8a8bc122113e26ad is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } })
1214 20:52:16.667838 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:16.667861 (+ 23us) write_op.cc:454] Released schema lock
1214 20:52:16.667871 (+ 10us) write_op.cc:335] FINISH: Op aborted
1214 20:52:16.667976 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:16.667981 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":267348,"prepare.run_cpu_time_us":1865,"prepare.run_wall_time_us":653904,"spinlock_wait_cycles":1499361536}]]}
I20251214 20:52:16.700184 1206 consensus_queue.cc:260] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.671715 522 rpcz_store.cc:276] 1214 20:52:15.640317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.762199 (+121882us) service_pool.cc:225] Handling call
1214 20:52:15.763048 (+ 849us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:15.763507 (+ 459us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:16.671589 (+908082us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:16.668119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:16.668264 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:16.668268 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:16.668270 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:16.668378 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:16.668390 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:16.668395 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:16.668488 (+ 93us) tablet.cc:699] Row locks acquired
1214 20:52:16.668491 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:16.668536 (+ 45us) op_driver.cc:389] HandleFailure(Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.670s ago.)
1214 20:52:16.668544 (+ 8us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:16.668554 (+ 10us) write_op.cc:454] Released schema lock
1214 20:52:16.668558 (+ 4us) write_op.cc:335] FINISH: Op aborted
1214 20:52:16.668621 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:16.668653 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":904120,"prepare.run_cpu_time_us":504,"prepare.run_wall_time_us":503}]]}
I20251214 20:52:16.707453 1206 ts_tablet_manager.cc:1434] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251214 20:52:16.708418 424 master_runner.cc:263] No leader master found from master c27224391721467a8f46e467a980b37a
I20251214 20:52:16.708534 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251214 20:52:16.710204 1661 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 20:52:16.711926 1206 tablet_bootstrap.cc:492] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.714673 1667 raft_consensus.cc:2804] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.715360 1203 ts_tablet_manager.cc:1434] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.077s user 0.006s sys 0.001s
I20251214 20:52:16.717856 1203 tablet_bootstrap.cc:492] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.719974 1650 raft_consensus.cc:493] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.720172 1650 raft_consensus.cc:515] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.720834 1650 leader_election.cc:290] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.721676 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3a2143b5508445492b970d2588cf1cf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.721753 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3a2143b5508445492b970d2588cf1cf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.721879 1206 tablet_bootstrap.cc:654] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.722178 1203 tablet_bootstrap.cc:654] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:16.722579 848 leader_election.cc:343] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:16.722851 851 leader_election.cc:343] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.722978 851 leader_election.cc:304] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:16.723449 1650 raft_consensus.cc:2749] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:16.725003 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]
I20251214 20:52:16.725632 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]
I20251214 20:52:16.726104 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_6838d68b72c042a293fd6c13096e6b78 [id=065328c700574078a1380f42b9848e2c]
I20251214 20:52:16.726579 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_616b5b16d08b43eb8a04a21a352031fe [id=086ee2aba6724a93b4bb736886fe9615]
I20251214 20:52:16.727440 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]
I20251214 20:52:16.727886 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_0870823e1f29409f9d61ebee0bbd060d [id=0a848dd5f6474712b2eba70add3743dc]
I20251214 20:52:16.727994 1203 tablet_bootstrap.cc:492] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.728181 1203 ts_tablet_manager.cc:1403] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20251214 20:52:16.728755 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_5242f3b282284ab7aff8525106732d53 [id=0d173bfba2b343dbac6a02a5ff84c828]
I20251214 20:52:16.729214 1650 raft_consensus.cc:493] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.729465 1650 raft_consensus.cc:515] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.729918 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu with pid 493
I20251214 20:52:16.729950 1203 raft_consensus.cc:359] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.730141 1203 raft_consensus.cc:385] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.730219 1203 raft_consensus.cc:740] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.730234 1650 leader_election.cc:290] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.730592 1667 raft_consensus.cc:697] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:16.731266 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd88d1e95ee04ef596863e1ae3687de0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.730916 1667 consensus_queue.cc:237] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.731946 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd88d1e95ee04ef596863e1ae3687de0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:16.732578 848 leader_election.cc:343] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:16.733126 989 leader_election.cc:343] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:16.733285 851 leader_election.cc:343] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.733408 851 leader_election.cc:304] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:16.733902 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d768c45bc02b4d108798efbf3f6aaf7d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.734273 1650 raft_consensus.cc:2749] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:16.734992 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]
I20251214 20:52:16.735486 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_5de120dfaab44b84868db0d1f79e2cee [id=1158f8a25d2b4696a5f5ed2c3a426862]
I20251214 20:52:16.737187 1206 tablet_bootstrap.cc:492] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
W20251214 20:52:16.737810 989 leader_election.cc:343] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.738169 989 leader_election.cc:304] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.739497 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_ee7f67ff1a1c4014a1cc4f9cf7f6613b [id=1597e24e3fff4007ab66b44505c960d8]
I20251214 20:52:16.739894 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c]
I20251214 20:52:16.739799 1588 raft_consensus.cc:2749] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:16.741394 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_26795454cb054bd6bdb4100765bd3c17 [id=1a3349421ffd4a7d933c8301fffb9154]
I20251214 20:52:16.741428 1203 consensus_queue.cc:260] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.741765 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_ba612520b0be474db6573ce754044c8a [id=1bf28f656cfe496d8accd428d0b238f2]
I20251214 20:52:16.742189 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_250b3222a83c44b08856444499fa612d [id=1ca50be529bd48398643cb6882958f77]
I20251214 20:52:16.742391 1203 ts_tablet_manager.cc:1434] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.014s user 0.002s sys 0.002s
I20251214 20:52:16.742609 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_501d1ae22d314c82914be72476d1ee0d [id=1f1f307f926f4ce297cf3f9c5423e818]
I20251214 20:52:16.743439 1203 tablet_bootstrap.cc:492] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.744633 1206 ts_tablet_manager.cc:1403] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20251214 20:52:16.745568 1206 raft_consensus.cc:359] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.745788 1206 raft_consensus.cc:385] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.745954 1206 raft_consensus.cc:740] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.746272 1206 consensus_queue.cc:260] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.747462 1203 tablet_bootstrap.cc:654] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.749258 1206 ts_tablet_manager.cc:1434] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:16.750749 1206 tablet_bootstrap.cc:492] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.752977 1206 tablet_bootstrap.cc:654] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.756304 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3]
I20251214 20:52:16.756982 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 [id=23efc0ee36244ca0a6ff0f2faeb00b1f]
I20251214 20:52:16.760164 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_8076bc8b31ba411faf089f314e6ed2ef [id=25016f04bb864f92a1cdfb6ceaa9fba9]
I20251214 20:52:16.760857 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_4907bb640d114586b98e6fa0e514cb5a [id=2a6d9d4658f24eb5b39adea101a0891c]
I20251214 20:52:16.761731 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_431bf74dfb32400bb0b2ef098c18e8df [id=2fcb81bfc6904fdd924de2494aba2e4a]
I20251214 20:52:16.761801 1667 raft_consensus.cc:493] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.762511 1203 tablet_bootstrap.cc:492] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.762663 1203 ts_tablet_manager.cc:1403] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20251214 20:52:16.762290 1667 raft_consensus.cc:515] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.763434 1203 raft_consensus.cc:359] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.763608 1203 raft_consensus.cc:385] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.763688 1203 raft_consensus.cc:740] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.764766 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a142157acbcd44adb6099b4428103ee5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:16.764961 915 raft_consensus.cc:2468] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:16.765347 991 leader_election.cc:304] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.765841 1588 raft_consensus.cc:2804] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.766104 1588 raft_consensus.cc:493] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.766346 1588 raft_consensus.cc:3060] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.763542 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_6a61fe3bf1734a888048db0d20abac60 [id=2fd12be2afd34815aff50d4ba78ea3f1]
I20251214 20:52:16.768951 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]
I20251214 20:52:16.770917 1203 consensus_queue.cc:260] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.771823 1203 ts_tablet_manager.cc:1434] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:16.772536 1203 tablet_bootstrap.cc:492] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.773614 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]
I20251214 20:52:16.774209 1203 tablet_bootstrap.cc:654] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.774721 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_b92e50faaffb45c0a3e1c7a94166f985 [id=406157031d694ada8a9fcba4e4e23e90]
I20251214 20:52:16.775444 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]
I20251214 20:52:16.776861 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_c845c10eeec54a7aa19cb8e0cebeb3a6 [id=4329d01e7f6147d0adad53fa86369b17]
I20251214 20:52:16.777979 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a142157acbcd44adb6099b4428103ee5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:16.778627 989 leader_election.cc:343] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.778239 1667 leader_election.cc:290] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.779592 1588 raft_consensus.cc:515] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.780113 1203 tablet_bootstrap.cc:492] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.780788 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_e82b745b92174d32a0d7bc8f779a6558 [id=43a932a17bd545f282a790e6c094a4e6]
I20251214 20:52:16.781543 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a142157acbcd44adb6099b4428103ee5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:16.781762 915 raft_consensus.cc:3060] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.780268 1203 ts_tablet_manager.cc:1403] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20251214 20:52:16.782636 1203 raft_consensus.cc:359] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.782792 1203 raft_consensus.cc:385] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.782871 1203 raft_consensus.cc:740] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.784435 915 raft_consensus.cc:2468] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:16.784659 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]
I20251214 20:52:16.785130 991 leader_election.cc:304] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.785692 1667 raft_consensus.cc:2804] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.786300 1667 raft_consensus.cc:697] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:16.786578 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]
I20251214 20:52:16.786866 1667 consensus_queue.cc:237] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.787950 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a142157acbcd44adb6099b4428103ee5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:16.787899 1588 leader_election.cc:290] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.788429 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_0dcbd217695546df9d1ca32583980907 [id=487c7f5559f64d13ae2de3da7c3e49a9]
W20251214 20:52:16.788565 989 leader_election.cc:343] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.789194 1203 consensus_queue.cc:260] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.790163 1203 ts_tablet_manager.cc:1434] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251214 20:52:16.790534 1203 tablet_bootstrap.cc:492] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.791512 1206 tablet_bootstrap.cc:492] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.791723 1206 ts_tablet_manager.cc:1403] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.003s
I20251214 20:52:16.792122 1203 tablet_bootstrap.cc:654] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.793090 1206 raft_consensus.cc:359] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.793351 1206 raft_consensus.cc:385] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.793498 1206 raft_consensus.cc:740] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.793794 1206 consensus_queue.cc:260] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.796770 1206 ts_tablet_manager.cc:1434] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:16.797149 1206 tablet_bootstrap.cc:492] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.797500 1202 tablet_bootstrap.cc:492] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.797686 1202 ts_tablet_manager.cc:1403] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.133s user 0.023s sys 0.020s
I20251214 20:52:16.797879 1203 tablet_bootstrap.cc:492] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.798018 1203 ts_tablet_manager.cc:1403] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20251214 20:52:16.798460 1202 raft_consensus.cc:359] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.798662 1202 raft_consensus.cc:385] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.798729 1202 raft_consensus.cc:740] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.798754 1203 raft_consensus.cc:359] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.798946 1203 raft_consensus.cc:385] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.799044 1203 raft_consensus.cc:740] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.799070 1202 consensus_queue.cc:260] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.799782 1202 ts_tablet_manager.cc:1434] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:16.800134 1202 tablet_bootstrap.cc:492] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.801587 1203 consensus_queue.cc:260] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.802183 1203 ts_tablet_manager.cc:1434] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:16.802495 1203 tablet_bootstrap.cc:492] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.802502 1202 tablet_bootstrap.cc:654] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.805295 1203 tablet_bootstrap.cc:654] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.809481 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]
I20251214 20:52:16.810463 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]
I20251214 20:52:16.811529 1206 tablet_bootstrap.cc:654] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.816504 1588 raft_consensus.cc:493] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.816737 1588 raft_consensus.cc:515] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.817643 1588 leader_election.cc:290] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.817428 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_12d421ed905d41eead75b19eb8a793b6 [id=5272550343144d4baaba07e97ebaa187]
I20251214 20:52:16.818902 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25974e5db555447baf5799775d15cd14" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.818895 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25974e5db555447baf5799775d15cd14" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:16.819151 774 raft_consensus.cc:2468] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:16.819185 915 raft_consensus.cc:2468] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:16.819619 991 leader_election.cc:304] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.819857 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]
I20251214 20:52:16.820286 1588 raft_consensus.cc:2804] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.820312 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_a40d3fbe67ba492d96c7db49c759836e [id=5308111e52e6484fbf050a603a18a3d7]
I20251214 20:52:16.820406 1588 raft_consensus.cc:493] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.820498 1588 raft_consensus.cc:3060] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.821332 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_7d6d418a1acb404f908e9ee299cef249 [id=55fc2a6c5e514513904682dfde510d9f]
I20251214 20:52:16.822207 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1]
I20251214 20:52:16.822868 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_55706259657e4a09aa794297547e9d98 [id=59d9716c5cee42c391018111323c5181]
I20251214 20:52:16.823433 1588 raft_consensus.cc:515] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.824487 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_b0d8d299bd0b4950948be859b03a3731 [id=5ded9fe825ae4308b5bd4b21c16634c8]
I20251214 20:52:16.824735 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25974e5db555447baf5799775d15cd14" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:16.824990 915 raft_consensus.cc:3060] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.825659 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]
I20251214 20:52:16.826890 1203 tablet_bootstrap.cc:492] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.827056 1203 ts_tablet_manager.cc:1403] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.004s
I20251214 20:52:16.827891 915 raft_consensus.cc:2468] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:16.827903 1203 raft_consensus.cc:359] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.828080 1203 raft_consensus.cc:385] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.828176 1203 raft_consensus.cc:740] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.828380 991 leader_election.cc:304] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:16.828398 1203 consensus_queue.cc:260] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.828783 1595 raft_consensus.cc:2804] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.828939 1203 ts_tablet_manager.cc:1434] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:16.828987 1595 raft_consensus.cc:697] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:16.829254 1203 tablet_bootstrap.cc:492] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.829293 1595 consensus_queue.cc:237] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.829483 1588 leader_election.cc:290] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.830509 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25974e5db555447baf5799775d15cd14" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:16.830708 774 raft_consensus.cc:3060] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.830792 1202 tablet_bootstrap.cc:492] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.830920 1202 ts_tablet_manager.cc:1403] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.003s
I20251214 20:52:16.833678 774 raft_consensus.cc:2468] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:16.835723 1206 tablet_bootstrap.cc:492] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.835961 1206 ts_tablet_manager.cc:1403] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.001s
I20251214 20:52:16.837677 1202 raft_consensus.cc:359] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.837895 1202 raft_consensus.cc:385] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.837996 1202 raft_consensus.cc:740] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.837759 1206 raft_consensus.cc:359] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.838284 1206 raft_consensus.cc:385] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.838227 1202 consensus_queue.cc:260] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.838424 1206 raft_consensus.cc:740] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.838694 1202 ts_tablet_manager.cc:1434] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20251214 20:52:16.839075 1202 tablet_bootstrap.cc:492] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.840891 1202 tablet_bootstrap.cc:654] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.843142 1476 raft_consensus.cc:493] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.843346 1476 raft_consensus.cc:515] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.844048 1476 leader_election.cc:290] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:16.844838 712 leader_election.cc:343] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 63497e77baeb4d328be2421b5732a28a
I20251214 20:52:16.838732 1206 consensus_queue.cc:260] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.846925 1203 tablet_bootstrap.cc:654] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.847352 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63497e77baeb4d328be2421b5732a28a" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:16.848819 713 leader_election.cc:343] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.848995 713 leader_election.cc:304] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:16.851567 1476 raft_consensus.cc:2749] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:16.852891 1206 ts_tablet_manager.cc:1434] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20251214 20:52:16.857517 1206 tablet_bootstrap.cc:492] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.857774 1202 tablet_bootstrap.cc:492] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.857973 1202 ts_tablet_manager.cc:1403] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.006s
I20251214 20:52:16.858320 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_55fe32e16f364e58afdd57c80a8966ae [id=6157817e96d642d09f9941ad7f7e82ee]
I20251214 20:52:16.858800 1202 raft_consensus.cc:359] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:16.859069 1202 raft_consensus.cc:385] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.859191 1202 raft_consensus.cc:740] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.859552 1202 consensus_queue.cc:260] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:16.859738 1650 raft_consensus.cc:493] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.859886 1650 raft_consensus.cc:515] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:16.860240 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_59405cb9326245918d9cf6fb65ddbf40 [id=619272205b724c56ba3d2542c2c5879a]
I20251214 20:52:16.860508 1650 leader_election.cc:290] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.860975 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_2f947b9f382942e082ea4909657a8f34 [id=66d124c2a8dd4cc188a832eff1dbe9ed]
I20251214 20:52:16.861313 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b9ca42a5ce4572aebe2a24f4758b12" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:16.862257 851 leader_election.cc:343] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.862886 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b9ca42a5ce4572aebe2a24f4758b12" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.863260 1206 tablet_bootstrap.cc:654] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:16.863483 848 leader_election.cc:343] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.863615 848 leader_election.cc:304] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:16.864097 1650 raft_consensus.cc:2749] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:16.864907 1202 ts_tablet_manager.cc:1434] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20251214 20:52:16.866407 1202 tablet_bootstrap.cc:492] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.868216 1202 tablet_bootstrap.cc:654] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.874670 1203 tablet_bootstrap.cc:492] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.875618 1203 ts_tablet_manager.cc:1403] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.002s
I20251214 20:52:16.876096 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_00bef63ea36c4a7585c2832be1454b68 [id=6a279e57008549c593df838211cc9cc1]
I20251214 20:52:16.880159 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7]
I20251214 20:52:16.884405 1203 raft_consensus.cc:359] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.884640 1203 raft_consensus.cc:385] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.884711 1203 raft_consensus.cc:740] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.884857 1202 tablet_bootstrap.cc:492] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.885030 1202 ts_tablet_manager.cc:1403] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.004s
I20251214 20:52:16.884981 1203 consensus_queue.cc:260] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.885504 1203 ts_tablet_manager.cc:1434] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251214 20:52:16.885779 1202 raft_consensus.cc:359] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:16.885972 1202 raft_consensus.cc:385] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.886070 1202 raft_consensus.cc:740] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.886328 1202 consensus_queue.cc:260] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:16.886976 1202 ts_tablet_manager.cc:1434] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:16.893533 1203 tablet_bootstrap.cc:492] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.894973 1202 tablet_bootstrap.cc:492] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.896840 1202 tablet_bootstrap.cc:654] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.897082 1203 tablet_bootstrap.cc:654] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.901962 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_1917701c01bf48c7b2bfc335d05f6db0 [id=6fba2831971a4bffac786611b54a6468]
W20251214 20:52:16.902132 992 connection.cc:537] client connection to 127.0.104.61:34819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251214 20:52:16.902771 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]
I20251214 20:52:16.903919 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_b151b87a38944a7a89d0fe6ff10ae68b [id=7b72926c96404fa5b84cfb86e6ae10a7]
I20251214 20:52:16.905324 1206 tablet_bootstrap.cc:492] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.905583 1206 ts_tablet_manager.cc:1403] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.001s
I20251214 20:52:16.906284 1203 tablet_bootstrap.cc:492] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.906451 1203 ts_tablet_manager.cc:1403] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20251214 20:52:16.906659 1206 raft_consensus.cc:359] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.906906 1206 raft_consensus.cc:385] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.907033 1206 raft_consensus.cc:740] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.907371 1206 consensus_queue.cc:260] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.907411 1203 raft_consensus.cc:359] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.907601 1203 raft_consensus.cc:385] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.907704 1203 raft_consensus.cc:740] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.907948 1203 consensus_queue.cc:260] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.908867 1202 tablet_bootstrap.cc:492] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.909188 1202 ts_tablet_manager.cc:1403] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251214 20:52:16.911031 1202 raft_consensus.cc:359] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:16.911427 1202 raft_consensus.cc:385] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.911557 1202 raft_consensus.cc:740] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.911716 1203 ts_tablet_manager.cc:1434] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:16.912027 1203 tablet_bootstrap.cc:492] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.914109 1650 raft_consensus.cc:493] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.914274 1650 raft_consensus.cc:515] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.914889 1650 leader_election.cc:290] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.915634 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a9a6192c744282ab5672cbe15fd299" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:16.916224 848 leader_election.cc:343] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.917469 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]
I20251214 20:52:16.919885 1206 ts_tablet_manager.cc:1434] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251214 20:52:16.920908 1203 tablet_bootstrap.cc:654] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.921569 1650 raft_consensus.cc:493] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.921723 1650 raft_consensus.cc:515] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.922339 1206 tablet_bootstrap.cc:492] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.922362 1650 leader_election.cc:290] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.924067 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a9a6192c744282ab5672cbe15fd299" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.925282 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d74d3abc45a41969289c0a0892bfa04" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.925448 1054 raft_consensus.cc:2468] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:16.925892 851 leader_election.cc:343] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.926043 851 leader_election.cc:304] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:16.923149 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d74d3abc45a41969289c0a0892bfa04" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:16.926157 1202 consensus_queue.cc:260] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:16.926915 848 leader_election.cc:343] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.926936 1653 raft_consensus.cc:2749] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:16.927538 851 leader_election.cc:304] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:16.927982 1650 raft_consensus.cc:2804] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.928112 1650 raft_consensus.cc:493] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.928203 1650 raft_consensus.cc:3060] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.929203 1203 tablet_bootstrap.cc:492] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:16.929394 1203 ts_tablet_manager.cc:1403] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20251214 20:52:16.929536 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_180ac2f089ba44298206726a47570c5b [id=81609d89fec84b2f9b79ca8986447b65]
I20251214 20:52:16.929967 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_dbcc1409fc7e48cb804842a37eebaad2 [id=81c0278b4add47d9947b461866fe9eae]
I20251214 20:52:16.930401 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]
I20251214 20:52:16.930333 1203 raft_consensus.cc:359] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.930550 1203 raft_consensus.cc:385] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.930626 1203 raft_consensus.cc:740] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.930811 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_e5f3db562155448e91e2e4314254cbe6 [id=826a2666d167435c8fdfd9ab8b8819e6]
I20251214 20:52:16.930866 1650 raft_consensus.cc:515] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.930902 1203 consensus_queue.cc:260] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.931950 1202 ts_tablet_manager.cc:1434] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20251214 20:52:16.932760 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d74d3abc45a41969289c0a0892bfa04" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:16.933320 848 leader_election.cc:343] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:16.933804 1650 leader_election.cc:290] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:16.934435 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d74d3abc45a41969289c0a0892bfa04" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:16.934626 1054 raft_consensus.cc:3060] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.935168 1206 tablet_bootstrap.cc:654] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.936584 1203 ts_tablet_manager.cc:1434] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20251214 20:52:16.937444 1054 raft_consensus.cc:2468] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:16.937476 1202 tablet_bootstrap.cc:492] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:16.937958 851 leader_election.cc:304] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:16.938457 1650 raft_consensus.cc:2804] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:16.938648 1650 raft_consensus.cc:697] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:16.939180 1650 consensus_queue.cc:237] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:16.941145 1203 tablet_bootstrap.cc:492] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:16.943207 1203 tablet_bootstrap.cc:654] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.944733 1202 tablet_bootstrap.cc:654] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:16.946857 851 connection.cc:537] client connection to 127.0.104.61:34819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251214 20:52:16.947955 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_893229542bec44e8b903bb0934bd16f5 [id=83b21f4baa574636ae3b50358fefae95]
I20251214 20:52:16.948755 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_13e60d7ade27439bbcab22ee125a50ba [id=8789a018b1c24bc3956fd11aea52b20e]
I20251214 20:52:16.950285 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_bb6020a9d77b42b5955933a449edc238 [id=8872bfac95c64cbba1ccb84d1d9f2483]
I20251214 20:52:16.951403 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_6a0720013c0d470e91997a7ad8d483fd [id=88d5807cc76646e3a2962b5d36413894]
I20251214 20:52:16.954208 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_6bac3fed78714367b43db88be8ee1712 [id=8cd62da3d57c4be4b5f10ace49d2e148]
I20251214 20:52:16.954667 1476 raft_consensus.cc:493] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:16.954886 1476 raft_consensus.cc:515] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.956065 1476 leader_election.cc:290] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:16.956916 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d768c45bc02b4d108798efbf3f6aaf7d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:16.957355 1054 raft_consensus.cc:2468] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:16.957906 713 leader_election.cc:304] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:16.958310 1476 raft_consensus.cc:2804] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:16.958406 1476 raft_consensus.cc:493] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:16.958494 1476 raft_consensus.cc:3060] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.958510 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f]
I20251214 20:52:16.958967 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_3d23e6dc5242452693b8aaac124a863f [id=8f720514e13b45689817a9811fb624d9]
I20251214 20:52:16.959424 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_181aab73cbf1461b85364fc893700db7 [id=8fd9be1206a74aa1921e0dade8be3774]
I20251214 20:52:16.959874 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]
W20251214 20:52:16.960114 712 leader_election.cc:343] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: d768c45bc02b4d108798efbf3f6aaf7d
I20251214 20:52:16.960459 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7]
W20251214 20:52:16.960654 713 connection.cc:537] client connection to 127.0.104.61:34819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251214 20:52:16.961117 1476 raft_consensus.cc:515] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.962574 1476 leader_election.cc:290] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:16.962836 1690 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 20:52:16.903154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:34819 (local address 127.0.104.3:55649)
1214 20:52:16.923295 (+ 20141us) negotiation.cc:107] Waiting for socket to connect
1214 20:52:16.923324 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1214 20:52:16.932976 (+ 9652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 20:52:16.961716 (+ 28740us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":19951,"thread_start_us":20338,"threads_started":1}
W20251214 20:52:16.966552 712 leader_election.cc:343] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: d768c45bc02b4d108798efbf3f6aaf7d
I20251214 20:52:16.971318 1334 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 4 FOLLOWER]: Refusing update from remote peer c27224391721467a8f46e467a980b37a: Log matching property violated. Preceding OpId in replica: term: 3 index: 423. Preceding OpId from leader: term: 4 index: 424. (index mismatch)
W20251214 20:52:16.972879 1695 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 20:52:16.947784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:34819 (local address 127.0.104.2:47719)
1214 20:52:16.959085 (+ 11301us) negotiation.cc:107] Waiting for socket to connect
1214 20:52:16.959109 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1214 20:52:16.959281 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1214 20:52:16.972383 (+ 13102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":11159,"thread_start_us":11582,"threads_started":1}
I20251214 20:52:16.976140 1206 tablet_bootstrap.cc:492] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:16.976497 1206 ts_tablet_manager.cc:1403] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.000s
I20251214 20:52:16.977612 1206 raft_consensus.cc:359] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.978737 1206 raft_consensus.cc:385] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.978922 1206 raft_consensus.cc:740] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.979403 1206 consensus_queue.cc:260] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:16.983662 1206 ts_tablet_manager.cc:1434] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:16.984143 1206 tablet_bootstrap.cc:492] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:16.988786 1206 tablet_bootstrap.cc:654] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:16.960826 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_2ae9b9f412384e96bfa1e3e6d26ea104 [id=9553b46856c94003a293135c814a7df7]
I20251214 20:52:16.972092 1659 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 424, Last known committed idx: 423, Time since last communication: 0.019s
I20251214 20:52:16.991365 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d768c45bc02b4d108798efbf3f6aaf7d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:16.991611 1054 raft_consensus.cc:3060] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:16.992782 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d]
I20251214 20:52:16.993148 1655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27224391721467a8f46e467a980b37a. Latest consensus state: current_term: 4 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:16.993427 1655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:16.993726 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]
I20251214 20:52:16.994338 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_5275181d46dc4a85b27fc0c45fa587fe [id=ab9838348eb24201bcb2866b16994965]
I20251214 20:52:16.994379 1054 raft_consensus.cc:2468] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:16.996325 1202 tablet_bootstrap.cc:492] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:16.996591 1202 ts_tablet_manager.cc:1403] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.008s
I20251214 20:52:16.998127 1202 raft_consensus.cc:359] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:16.998464 1202 raft_consensus.cc:385] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:16.998653 1202 raft_consensus.cc:740] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:16.999081 1202 consensus_queue.cc:260] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.000069 1202 ts_tablet_manager.cc:1434] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251214 20:52:17.000624 1202 tablet_bootstrap.cc:492] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.004500 1202 tablet_bootstrap.cc:654] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.006604 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]
I20251214 20:52:17.010515 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_9ad469dea994477983bfe62eef37e334 [id=b1be575d16354425ab4f7e8e62847820]
I20251214 20:52:17.015092 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_cf7fd4bd0b764cc79acc374ea7bb5187 [id=b3df2580f4a74189aa06f021d6b4ce71]
I20251214 20:52:17.015167 1203 tablet_bootstrap.cc:492] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.015327 1203 ts_tablet_manager.cc:1403] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.074s user 0.016s sys 0.032s
I20251214 20:52:17.016352 1203 raft_consensus.cc:359] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.016571 1203 raft_consensus.cc:385] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.016654 1203 raft_consensus.cc:740] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.017264 1203 consensus_queue.cc:260] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.018585 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]
I20251214 20:52:17.019796 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_b3f5ab40d9334de1a194aab3347387d7 [id=b470f1b24a6d4e7ab4420ae212e54ff0]
I20251214 20:52:17.023164 1203 ts_tablet_manager.cc:1434] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251214 20:52:17.027697 713 leader_election.cc:304] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.028343 1476 raft_consensus.cc:2804] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.028505 1476 raft_consensus.cc:697] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:17.028834 1476 consensus_queue.cc:237] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.034111 1655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:17.034359 1655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:17.037894 1465 consensus_queue.cc:1048] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.042040 1206 tablet_bootstrap.cc:492] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:17.042951 1203 tablet_bootstrap.cc:492] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.045737 1203 tablet_bootstrap.cc:654] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.023206 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_dd97baf658ea4f44bba79f081b5ff75e [id=ba545f2c5ba949709f18c486d993faab]
I20251214 20:52:17.049485 1206 ts_tablet_manager.cc:1403] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.001s
I20251214 20:52:17.050875 1206 raft_consensus.cc:359] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.051954 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_642da83f7ed0444688afdf91b7a9e7e8 [id=bca3a2d37f8b4fc797a5c52a43c14869]
I20251214 20:52:17.052481 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]
I20251214 20:52:17.054337 1206 raft_consensus.cc:385] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.055720 1206 raft_consensus.cc:740] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.054724 1659 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:17.055992 1659 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:17.056393 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_67ec8d0cd3f948218b4637de6d68a57b [id=bdc2d86a977d4146a1a86717c5916807]
I20251214 20:52:17.056895 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]
I20251214 20:52:17.057281 1476 raft_consensus.cc:493] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.057427 1476 raft_consensus.cc:515] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.058461 1476 leader_election.cc:290] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:17.059046 712 leader_election.cc:343] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: f0d661a1bb034053a4634ff537eafc7d
I20251214 20:52:17.059528 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0d661a1bb034053a4634ff537eafc7d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.059747 1054 raft_consensus.cc:2468] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
W20251214 20:52:17.060415 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251214 20:52:17.063989 1476 consensus_queue.cc:1048] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.064985 1202 tablet_bootstrap.cc:492] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.065387 1202 ts_tablet_manager.cc:1403] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.065s user 0.017s sys 0.006s
I20251214 20:52:17.065760 1203 tablet_bootstrap.cc:492] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.065937 1203 ts_tablet_manager.cc:1403] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.003s
I20251214 20:52:17.066834 1203 raft_consensus.cc:359] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.066838 1202 raft_consensus.cc:359] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.067061 1203 raft_consensus.cc:385] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.067152 1203 raft_consensus.cc:740] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.067219 1202 raft_consensus.cc:385] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.067364 1202 raft_consensus.cc:740] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.067934 1202 consensus_queue.cc:260] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.069434 1203 consensus_queue.cc:260] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.070052 1202 ts_tablet_manager.cc:1434] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251214 20:52:17.071239 1202 tablet_bootstrap.cc:492] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.075484 713 leader_election.cc:304] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.075961 1203 ts_tablet_manager.cc:1434] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20251214 20:52:17.076110 1476 raft_consensus.cc:2804] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.076272 1476 raft_consensus.cc:493] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.076431 1476 raft_consensus.cc:3060] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.077586 1202 tablet_bootstrap.cc:654] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.080652 1203 tablet_bootstrap.cc:492] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.083583 1203 tablet_bootstrap.cc:654] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.086114 1476 raft_consensus.cc:515] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:17.088191 712 leader_election.cc:343] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: f0d661a1bb034053a4634ff537eafc7d
I20251214 20:52:17.089179 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f0d661a1bb034053a4634ff537eafc7d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:17.089439 1056 raft_consensus.cc:3060] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.092484 1056 raft_consensus.cc:2468] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:17.093134 713 leader_election.cc:304] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.094135 1206 consensus_queue.cc:260] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.095294 1714 raft_consensus.cc:2804] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.095983 1202 tablet_bootstrap.cc:492] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.096146 1202 ts_tablet_manager.cc:1403] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.004s
I20251214 20:52:17.096344 1206 ts_tablet_manager.cc:1434] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.046s user 0.002s sys 0.001s
I20251214 20:52:17.097028 1202 raft_consensus.cc:359] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.097251 1202 raft_consensus.cc:385] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.097337 1202 raft_consensus.cc:740] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.097568 1202 consensus_queue.cc:260] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.098460 1202 ts_tablet_manager.cc:1434] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:17.098886 1202 tablet_bootstrap.cc:492] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.101100 1202 tablet_bootstrap.cc:654] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.102648 1206 tablet_bootstrap.cc:492] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:17.106997 1202 tablet_bootstrap.cc:492] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.107218 1202 ts_tablet_manager.cc:1403] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20251214 20:52:17.108027 1202 raft_consensus.cc:359] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.108234 1202 raft_consensus.cc:385] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.108316 1202 raft_consensus.cc:740] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.108613 1202 consensus_queue.cc:260] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.109849 1202 ts_tablet_manager.cc:1434] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:17.110155 1202 tablet_bootstrap.cc:492] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.112303 1202 tablet_bootstrap.cc:654] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.115072 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]
I20251214 20:52:17.116040 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_8b86c42f7826440982c88a8625559146 [id=c35009dabd4a42598506e52b3d3e493c]
I20251214 20:52:17.116581 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]
I20251214 20:52:17.118295 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_53273de8ee634ce9994c46fe98bba85e [id=cd14def069aa430994cd391ddab6c4dd]
I20251214 20:52:17.121037 1202 tablet_bootstrap.cc:492] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.121243 1202 ts_tablet_manager.cc:1403] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20251214 20:52:17.122148 1202 raft_consensus.cc:359] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.122371 1202 raft_consensus.cc:385] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.122476 1202 raft_consensus.cc:740] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.122795 1202 consensus_queue.cc:260] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.123850 1202 ts_tablet_manager.cc:1434] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251214 20:52:17.124279 1202 tablet_bootstrap.cc:492] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.107663 1714 raft_consensus.cc:697] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:17.126792 1202 tablet_bootstrap.cc:654] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.126616 1714 consensus_queue.cc:237] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.134109 1206 tablet_bootstrap.cc:654] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.124799 1476 leader_election.cc:290] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.137322 1202 tablet_bootstrap.cc:492] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.137639 1202 ts_tablet_manager.cc:1403] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20251214 20:52:17.138659 1202 raft_consensus.cc:359] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.138964 1202 raft_consensus.cc:385] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.139081 1202 raft_consensus.cc:740] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.139607 1202 consensus_queue.cc:260] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.140619 1202 ts_tablet_manager.cc:1434] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:17.140974 1202 tablet_bootstrap.cc:492] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.143419 1715 consensus_queue.cc:1048] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.144001 1202 tablet_bootstrap.cc:654] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:17.148038 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:34819, user_credentials={real_user=slave}} blocked reactor thread for 87580.1us
W20251214 20:52:17.151494 442 consensus_peers.cc:597] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a -> Peer e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819): Couldn't send request to peer e6713a20a82b434f8a8bc122113e26ad. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:17.154438 1709 consensus_queue.cc:1048] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.154942 1465 consensus_queue.cc:1048] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 20:52:17.159464 1650 raft_consensus.cc:493] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.159713 1650 raft_consensus.cc:515] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.160470 1650 leader_election.cc:290] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.161350 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52daf1204bf142e7b82852bdec6582fb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.161734 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52daf1204bf142e7b82852bdec6582fb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:17.162801 851 leader_election.cc:343] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:17.163347 848 leader_election.cc:343] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.163523 848 leader_election.cc:304] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.164391 1650 raft_consensus.cc:2749] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.164428 1465 raft_consensus.cc:493] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.165002 1465 raft_consensus.cc:515] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.166548 1465 leader_election.cc:290] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:17.167261 712 leader_election.cc:343] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: c25832da5c1644329e18653000f3330e
I20251214 20:52:17.167797 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c25832da5c1644329e18653000f3330e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.168058 1053 raft_consensus.cc:2468] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:17.169179 713 leader_election.cc:304] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.170063 1465 raft_consensus.cc:2804] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.170287 1465 raft_consensus.cc:493] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.170548 1465 raft_consensus.cc:3060] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.174866 1202 tablet_bootstrap.cc:492] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.175062 1202 ts_tablet_manager.cc:1403] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.003s
I20251214 20:52:17.175096 1465 raft_consensus.cc:515] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.176067 1202 raft_consensus.cc:359] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.176343 1202 raft_consensus.cc:385] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.176479 1202 raft_consensus.cc:740] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.177426 1206 tablet_bootstrap.cc:492] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:17.180076 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c25832da5c1644329e18653000f3330e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:17.180493 1055 raft_consensus.cc:3060] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.183107 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_c3fc0fa4ec254ff889443791279bb8af [id=d31b208e2e284575ad429878129a28f5]
I20251214 20:52:17.178985 1715 raft_consensus.cc:493] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.183669 1715 raft_consensus.cc:515] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.184439 1715 leader_election.cc:290] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.179102 1465 leader_election.cc:290] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:17.179536 712 leader_election.cc:343] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: c25832da5c1644329e18653000f3330e
I20251214 20:52:17.187508 1055 raft_consensus.cc:2468] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:17.188249 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3a2143b5508445492b970d2588cf1cf" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.188652 1055 raft_consensus.cc:2468] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:17.188851 713 leader_election.cc:304] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.190061 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3a2143b5508445492b970d2588cf1cf" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.190320 1715 raft_consensus.cc:2804] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.190338 911 raft_consensus.cc:2468] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:17.190476 1715 raft_consensus.cc:697] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:17.190757 1715 consensus_queue.cc:237] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:17.191037 712 consensus_peers.cc:597] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8faa7fd7315f485c82aee38bcf4cd8fc. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:17.191308 712 leader_election.cc:304] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.192178 1476 raft_consensus.cc:2804] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.192394 1476 raft_consensus.cc:493] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.192618 1476 raft_consensus.cc:3060] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.195459 1202 consensus_queue.cc:260] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.195565 1476 raft_consensus.cc:515] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.196240 1206 ts_tablet_manager.cc:1403] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.094s user 0.000s sys 0.005s
I20251214 20:52:17.196925 1202 ts_tablet_manager.cc:1434] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20251214 20:52:17.197168 1206 raft_consensus.cc:359] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.197402 1206 raft_consensus.cc:385] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.197490 1206 raft_consensus.cc:740] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.197636 1202 tablet_bootstrap.cc:492] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.197716 1206 consensus_queue.cc:260] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.198993 1203 tablet_bootstrap.cc:492] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.199683 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_6f66a268b27e4f9b8d09597224149553 [id=d36ff09def9741f8b440f7bdbfdc21a8]
I20251214 20:52:17.200166 1203 ts_tablet_manager.cc:1403] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.120s user 0.009s sys 0.027s
I20251214 20:52:17.201287 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_08488285c9094a9dacc1e489c50271b3 [id=d3b4eef101ec491485c1889f08de9992]
I20251214 20:52:17.201247 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3a2143b5508445492b970d2588cf1cf" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:17.201478 915 raft_consensus.cc:3060] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.202255 1203 raft_consensus.cc:359] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.202972 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]
I20251214 20:52:17.205214 1203 raft_consensus.cc:385] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.206360 1709 consensus_queue.cc:1048] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.207095 1202 tablet_bootstrap.cc:654] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.207113 1203 raft_consensus.cc:740] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.207695 1206 ts_tablet_manager.cc:1434] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251214 20:52:17.208366 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_005b061b84dc4e7aa3fe295da6b99cd1 [id=d6c5c9fbf30b4477b502d001a4f3cd27]
I20251214 20:52:17.208726 1203 consensus_queue.cc:260] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.209324 1206 tablet_bootstrap.cc:492] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:17.212477 1203 ts_tablet_manager.cc:1434] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251214 20:52:17.213141 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_5e8a4451754247c2add9a3245b63b050 [id=d7f40373ddad4286a8fe7966df2d9668]
I20251214 20:52:17.216261 1715 raft_consensus.cc:493] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.220245 1203 tablet_bootstrap.cc:492] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.221793 915 raft_consensus.cc:2468] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:17.221900 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_7bc0f81d22a643589e8a54ceb2477807 [id=db33edd24660463ea648be7da745e29c]
I20251214 20:52:17.224092 1714 raft_consensus.cc:493] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.221840 1650 raft_consensus.cc:493] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.224038 1715 raft_consensus.cc:515] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.224529 1650 raft_consensus.cc:515] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.225232 1650 leader_election.cc:290] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.225570 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589]
I20251214 20:52:17.226917 1714 raft_consensus.cc:515] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.227756 1714 leader_election.cc:290] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.228003 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3a2143b5508445492b970d2588cf1cf" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:17.227770 1715 leader_election.cc:290] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.228257 1054 raft_consensus.cc:3060] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.228932 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a142157acbcd44adb6099b4428103ee5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.229379 1727 raft_consensus.cc:493] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.230113 1203 tablet_bootstrap.cc:654] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.231666 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_88881a27865d4b0aa6db6a2b736f0c17 [id=e0d1675617c74aa8a8afd81bd90f9639]
I20251214 20:52:17.229558 1727 raft_consensus.cc:515] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.233574 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_3369f89218b140e7a52da8221726f4e1 [id=e901e2802c854b4f829e59e00aa90040]
I20251214 20:52:17.233717 1476 leader_election.cc:290] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.234354 1054 raft_consensus.cc:2468] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:17.234916 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a625fcb95324466a12ab118bdb859f1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.235252 1055 raft_consensus.cc:2468] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:17.235533 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_e532e5fa17f2476a90df03eb6449a067 [id=ece0780edb0f41b787b46b290c4c0515]
I20251214 20:52:17.235500 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023299180d074893847e84dcd9812751" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.235910 712 leader_election.cc:304] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.236922 1727 leader_election.cc:290] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.238698 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a625fcb95324466a12ab118bdb859f1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.238830 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bd6856b800247de896f16eee66f2827" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.239094 1056 raft_consensus.cc:2393] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate a2e8fd1cb731446d88f8e510c7a52999 in this term.
I20251214 20:52:17.239066 851 leader_election.cc:304] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.239418 1650 consensus_queue.cc:1048] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.239639 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]
I20251214 20:52:17.240142 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_ecba341e009147b9b843be246ea2cd7c [id=ed8d16aa8d984c4198c1d1b4ae3fc729]
W20251214 20:52:17.240111 848 leader_election.cc:343] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.240634 1650 raft_consensus.cc:2804] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.240773 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_b01c45a1fe8f4a71858fcb2127177471 [id=ef22180b36a241fea13d0e622c02b50a]
I20251214 20:52:17.240772 1650 raft_consensus.cc:493] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.240867 1650 raft_consensus.cc:3060] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.241176 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_f2360eda2fac4147840743a9a4f0bc2b [id=f1338e81f4ec4dce87130f2ddbe98041]
I20251214 20:52:17.241362 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a142157acbcd44adb6099b4428103ee5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.241675 915 raft_consensus.cc:2393] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:17.242156 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bd6856b800247de896f16eee66f2827" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.242480 1203 tablet_bootstrap.cc:492] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.242655 1203 ts_tablet_manager.cc:1403] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20251214 20:52:17.243369 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023299180d074893847e84dcd9812751" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.243613 911 raft_consensus.cc:2393] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:17.243690 1203 raft_consensus.cc:359] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.243871 1203 raft_consensus.cc:385] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.243870 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]
I20251214 20:52:17.243953 1203 raft_consensus.cc:740] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.244204 1203 consensus_queue.cc:260] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.244782 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_9e89276f2f2b4a7981f91242bdb24a87 [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]
I20251214 20:52:17.245223 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_cae5b2188af54d068db115a0745ee866 [id=f4157d7b9fc848a8b7fa08742b456167]
I20251214 20:52:17.245714 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]
I20251214 20:52:17.246682 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]
I20251214 20:52:17.247357 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc]
I20251214 20:52:17.247602 1650 raft_consensus.cc:515] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.249368 1650 leader_election.cc:290] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.249706 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_3736aabecd40417f9cb502ac612bacee [id=fc592ea105f242a299337c9cbe438778]
I20251214 20:52:17.250130 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a625fcb95324466a12ab118bdb859f1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:17.250376 1056 raft_consensus.cc:3060] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.250810 1202 tablet_bootstrap.cc:492] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.250986 1202 ts_tablet_manager.cc:1403] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.007s
I20251214 20:52:17.251832 1202 raft_consensus.cc:359] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.252029 1202 raft_consensus.cc:385] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.252101 1202 raft_consensus.cc:740] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.252362 1202 consensus_queue.cc:260] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.252900 1202 ts_tablet_manager.cc:1434] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:17.253129 1056 raft_consensus.cc:2468] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:17.254148 851 leader_election.cc:304] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.254765 1202 tablet_bootstrap.cc:492] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.255085 1653 raft_consensus.cc:2804] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.255344 1653 raft_consensus.cc:697] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:17.255695 1653 consensus_queue.cc:237] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.258005 712 leader_election.cc:304] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.258425 1202 tablet_bootstrap.cc:654] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.258534 712 leader_election.cc:304] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.260746 1661 catalog_manager.cc:679] Loaded metadata for table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]
I20251214 20:52:17.267482 1715 raft_consensus.cc:3060] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.268162 1727 raft_consensus.cc:3060] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.269557 1595 raft_consensus.cc:493] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.269763 1595 raft_consensus.cc:515] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.270560 1595 leader_election.cc:290] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.271636 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a9a6192c744282ab5672cbe15fd299" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.271914 911 raft_consensus.cc:2468] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.272643 991 leader_election.cc:304] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.273248 1595 raft_consensus.cc:2804] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.273365 1595 raft_consensus.cc:493] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.273451 1595 raft_consensus.cc:3060] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.276383 1595 raft_consensus.cc:515] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.276801 1650 raft_consensus.cc:493] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.277140 1202 tablet_bootstrap.cc:492] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.277132 1650 raft_consensus.cc:515] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.277552 1653 raft_consensus.cc:493] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.277760 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a9a6192c744282ab5672cbe15fd299" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:17.277557 1202 ts_tablet_manager.cc:1403] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.001s
I20251214 20:52:17.278215 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a625fcb95324466a12ab118bdb859f1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.278846 911 raft_consensus.cc:3060] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.279147 712 leader_election.cc:304] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.280272 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4626d02ca548f3bf5a1a9142600bff" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.280541 1650 leader_election.cc:290] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.281057 1717 consensus_queue.cc:1048] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
W20251214 20:52:17.280889 848 leader_election.cc:343] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.281790 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a9a6192c744282ab5672cbe15fd299" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.281435 1653 raft_consensus.cc:515] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.282150 774 raft_consensus.cc:2468] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
W20251214 20:52:17.282496 848 leader_election.cc:343] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.282979 1653 leader_election.cc:290] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.284742 911 raft_consensus.cc:2468] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.282512 1202 raft_consensus.cc:359] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.285398 1202 raft_consensus.cc:385] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.285616 1202 raft_consensus.cc:740] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.285888 1202 consensus_queue.cc:260] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.286393 991 leader_election.cc:304] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.286528 1202 ts_tablet_manager.cc:1434] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251214 20:52:17.286975 1727 raft_consensus.cc:2749] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.287042 1202 tablet_bootstrap.cc:492] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.287607 1661 tablet_loader.cc:96] loaded metadata for tablet 00cf456eda464f5ba22d6893f73b5cba (table default.Table_6838d68b72c042a293fd6c13096e6b78 [id=065328c700574078a1380f42b9848e2c])
I20251214 20:52:17.287626 1667 raft_consensus.cc:2804] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.287797 1667 raft_consensus.cc:697] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:17.288107 1667 consensus_queue.cc:237] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.288235 1661 tablet_loader.cc:96] loaded metadata for tablet 012b81223d0f4ae9b742014e329f11a9 (table default.Table_b151b87a38944a7a89d0fe6ff10ae68b [id=7b72926c96404fa5b84cfb86e6ae10a7])
I20251214 20:52:17.288604 1661 tablet_loader.cc:96] loaded metadata for tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151])
I20251214 20:52:17.288976 1661 tablet_loader.cc:96] loaded metadata for tablet 02e256d4537249a394b80dc36beefef4 (table default.Table_ecba341e009147b9b843be246ea2cd7c [id=ed8d16aa8d984c4198c1d1b4ae3fc729])
I20251214 20:52:17.288904 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60ae3db968f6412aa0c3952fafe65f88" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.289350 1661 tablet_loader.cc:96] loaded metadata for tablet 06381efe75284e649fd6d1a9903cfa9b (table default.Table_7bc0f81d22a643589e8a54ceb2477807 [id=db33edd24660463ea648be7da745e29c])
I20251214 20:52:17.289680 1661 tablet_loader.cc:96] loaded metadata for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_55fe32e16f364e58afdd57c80a8966ae [id=6157817e96d642d09f9941ad7f7e82ee])
I20251214 20:52:17.289966 1661 tablet_loader.cc:96] loaded metadata for tablet 082ec70b8ee646b898d9201ad073d292 (table default.Table_3d23e6dc5242452693b8aaac124a863f [id=8f720514e13b45689817a9811fb624d9])
I20251214 20:52:17.289994 1715 raft_consensus.cc:2749] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.290639 1717 raft_consensus.cc:3060] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.291172 1661 tablet_loader.cc:96] loaded metadata for tablet 0c973e0111804b8b869068bdcc5869bf (table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f])
I20251214 20:52:17.291555 1661 tablet_loader.cc:96] loaded metadata for tablet 0cafed2861e94cd9b0d10d33864ba45d (table default.Table_ba612520b0be474db6573ce754044c8a [id=1bf28f656cfe496d8accd428d0b238f2])
I20251214 20:52:17.291733 1476 raft_consensus.cc:2804] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.291956 1476 raft_consensus.cc:697] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:17.292495 1661 tablet_loader.cc:96] loaded metadata for tablet 140667afe42f42659a8cd38ed2517201 (table default.Table_13e60d7ade27439bbcab22ee125a50ba [id=8789a018b1c24bc3956fd11aea52b20e])
I20251214 20:52:17.292896 1661 tablet_loader.cc:96] loaded metadata for tablet 144dee1a6d7344f2b79da6e79e67599e (table default.Table_6bac3fed78714367b43db88be8ee1712 [id=8cd62da3d57c4be4b5f10ace49d2e148])
I20251214 20:52:17.293285 1661 tablet_loader.cc:96] loaded metadata for tablet 16816fa6b8c34207bb929147910b93cb (table default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39])
I20251214 20:52:17.293649 1661 tablet_loader.cc:96] loaded metadata for tablet 17550eb0ee6b476f8a8c304451f678b2 (table default.Table_6f66a268b27e4f9b8d09597224149553 [id=d36ff09def9741f8b440f7bdbfdc21a8])
I20251214 20:52:17.294147 1661 tablet_loader.cc:96] loaded metadata for tablet 18e119bc44874aecb4e83c8d6040c8cd (table default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1])
I20251214 20:52:17.294620 1661 tablet_loader.cc:96] loaded metadata for tablet 190afbb866854dd7b5393e04443752b7 (table default.Table_501d1ae22d314c82914be72476d1ee0d [id=1f1f307f926f4ce297cf3f9c5423e818])
I20251214 20:52:17.295022 1661 tablet_loader.cc:96] loaded metadata for tablet 1dca369dc5f54a4ea1783e334c5670d6 (table default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97])
I20251214 20:52:17.295506 1661 tablet_loader.cc:96] loaded metadata for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_53273de8ee634ce9994c46fe98bba85e [id=cd14def069aa430994cd391ddab6c4dd])
W20251214 20:52:17.296538 848 leader_election.cc:343] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.296700 1661 tablet_loader.cc:96] loaded metadata for tablet 242a77ae57174de48f4eaf43384ee0ba (table default.Table_9ad469dea994477983bfe62eef37e334 [id=b1be575d16354425ab4f7e8e62847820])
I20251214 20:52:17.297088 1476 consensus_queue.cc:237] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.297406 1661 tablet_loader.cc:96] loaded metadata for tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07])
I20251214 20:52:17.297827 1661 tablet_loader.cc:96] loaded metadata for tablet 25dcf491653147e7831217c1ec7a2e0c (table default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7])
I20251214 20:52:17.298204 1661 tablet_loader.cc:96] loaded metadata for tablet 274185a4115b458e8e9a376bdbcc870b (table default.Table_e82b745b92174d32a0d7bc8f779a6558 [id=43a932a17bd545f282a790e6c094a4e6])
I20251214 20:52:17.299491 1661 tablet_loader.cc:96] loaded metadata for tablet 2accf8a160584c82bb695a52e3f2b05c (table default.Table_c3fc0fa4ec254ff889443791279bb8af [id=d31b208e2e284575ad429878129a28f5])
I20251214 20:52:17.299939 1661 tablet_loader.cc:96] loaded metadata for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_616b5b16d08b43eb8a04a21a352031fe [id=086ee2aba6724a93b4bb736886fe9615])
I20251214 20:52:17.300365 1661 tablet_loader.cc:96] loaded metadata for tablet 2b92b6d5dde2465684d19aee44a83160 (table default.Table_29dd5d466c7d4c1698b88311145db749 [id=22d141506a1d401ca05343516b778cd3])
I20251214 20:52:17.300817 1661 tablet_loader.cc:96] loaded metadata for tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1])
I20251214 20:52:17.301239 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40a9a6192c744282ab5672cbe15fd299" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.301433 774 raft_consensus.cc:3060] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.301972 1661 tablet_loader.cc:96] loaded metadata for tablet 33a7f355b40f472883e3399209ef6762 (table default.Table_12d421ed905d41eead75b19eb8a793b6 [id=5272550343144d4baaba07e97ebaa187])
I20251214 20:52:17.302840 1202 tablet_bootstrap.cc:654] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.303444 1661 tablet_loader.cc:96] loaded metadata for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749])
I20251214 20:52:17.303812 1661 tablet_loader.cc:96] loaded metadata for tablet 3a23561b9f4e48bbbd796e14fafec086 (table default.Table_f2360eda2fac4147840743a9a4f0bc2b [id=f1338e81f4ec4dce87130f2ddbe98041])
I20251214 20:52:17.304291 1661 tablet_loader.cc:96] loaded metadata for tablet 3b47ca1a03944cf1847d3578856dafa7 (table default.Table_6a61fe3bf1734a888048db0d20abac60 [id=2fd12be2afd34815aff50d4ba78ea3f1])
I20251214 20:52:17.305071 774 raft_consensus.cc:2468] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.305766 1595 leader_election.cc:290] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.306701 1661 tablet_loader.cc:96] loaded metadata for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1])
I20251214 20:52:17.306902 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4626d02ca548f3bf5a1a9142600bff" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.294466 1717 raft_consensus.cc:2749] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.295657 1206 tablet_bootstrap.cc:654] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.307991 1661 tablet_loader.cc:96] loaded metadata for tablet 3fa0317e999b483788e5c5652e6e83de (table default.Table_b01c45a1fe8f4a71858fcb2127177471 [id=ef22180b36a241fea13d0e622c02b50a])
I20251214 20:52:17.308209 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60ae3db968f6412aa0c3952fafe65f88" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:17.310359 851 leader_election.cc:343] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.310518 851 leader_election.cc:304] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.310992 1735 raft_consensus.cc:2749] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.311983 1650 raft_consensus.cc:493] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.312208 1650 raft_consensus.cc:515] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:17.312709 851 leader_election.cc:343] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.312963 1650 leader_election.cc:290] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.312968 851 leader_election.cc:304] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.313385 1206 tablet_bootstrap.cc:492] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:17.313902 1650 raft_consensus.cc:2749] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.314414 1206 ts_tablet_manager.cc:1403] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.105s user 0.005s sys 0.000s
I20251214 20:52:17.314342 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b92b6d5dde2465684d19aee44a83160" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:17.315063 848 leader_election.cc:343] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.316077 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b92b6d5dde2465684d19aee44a83160" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.317041 1206 raft_consensus.cc:359] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
W20251214 20:52:17.317070 851 leader_election.cc:343] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.317483 851 leader_election.cc:304] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.317870 1650 raft_consensus.cc:2749] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.317997 1661 tablet_loader.cc:96] loaded metadata for tablet 42bdcecba5534294aca1eb419f18e9d2 (table default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8])
I20251214 20:52:17.318480 1661 tablet_loader.cc:96] loaded metadata for tablet 4408013cf49e46d0b18d24b48483ea49 (table default.Table_7d6d418a1acb404f908e9ee299cef249 [id=55fc2a6c5e514513904682dfde510d9f])
I20251214 20:52:17.320719 1661 tablet_loader.cc:96] loaded metadata for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3])
I20251214 20:52:17.322139 1661 tablet_loader.cc:96] loaded metadata for tablet 492770a7b8b74ba49315f974cc54eedb (table default.Table_005b061b84dc4e7aa3fe295da6b99cd1 [id=d6c5c9fbf30b4477b502d001a4f3cd27])
I20251214 20:52:17.322782 1661 tablet_loader.cc:96] loaded metadata for tablet 4a625fcb95324466a12ab118bdb859f1 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974])
I20251214 20:52:17.326864 1206 raft_consensus.cc:385] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.327098 1206 raft_consensus.cc:740] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.329078 1203 ts_tablet_manager.cc:1434] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.086s user 0.003s sys 0.000s
I20251214 20:52:17.327909 1206 consensus_queue.cc:260] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.330673 1206 ts_tablet_manager.cc:1434] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251214 20:52:17.331945 1661 tablet_loader.cc:96] loaded metadata for tablet 4aa1a44452074ec28a1bd9c48a5f47cb (table default.Table_dbcc1409fc7e48cb804842a37eebaad2 [id=81c0278b4add47d9947b461866fe9eae])
I20251214 20:52:17.332015 1203 tablet_bootstrap.cc:492] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.333655 1206 tablet_bootstrap.cc:492] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:17.333818 1661 tablet_loader.cc:96] loaded metadata for tablet 4d6e7306254044449f1127fb963ca5f0 (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e])
I20251214 20:52:17.334113 1595 raft_consensus.cc:493] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.334264 1661 tablet_loader.cc:96] loaded metadata for tablet 4f1d0e555eee4c56be271d14c34335bd (table default.Table_180ac2f089ba44298206726a47570c5b [id=81609d89fec84b2f9b79ca8986447b65])
I20251214 20:52:17.337190 1206 tablet_bootstrap.cc:654] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.337203 1661 tablet_loader.cc:96] loaded metadata for tablet 57f2314d33fa474ca36dafe0a0eae5ae (table default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c])
I20251214 20:52:17.337611 1661 tablet_loader.cc:96] loaded metadata for tablet 58ac06f852fc4abea5cb213926002ca1 (table default.Table_3369f89218b140e7a52da8221726f4e1 [id=e901e2802c854b4f829e59e00aa90040])
I20251214 20:52:17.334635 1595 raft_consensus.cc:515] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.338624 1661 tablet_loader.cc:96] loaded metadata for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4])
I20251214 20:52:17.339246 1650 raft_consensus.cc:493] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.339444 1650 raft_consensus.cc:515] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.339617 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd88d1e95ee04ef596863e1ae3687de0" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.339860 915 raft_consensus.cc:2468] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.340265 1650 leader_election.cc:290] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.340303 991 leader_election.cc:304] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.340764 1664 raft_consensus.cc:2804] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.340821 1661 tablet_loader.cc:96] loaded metadata for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7])
I20251214 20:52:17.340924 1664 raft_consensus.cc:493] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.340977 1595 leader_election.cc:290] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.341063 1664 raft_consensus.cc:3060] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.341204 1661 tablet_loader.cc:96] loaded metadata for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (table default.Table_9e89276f2f2b4a7981f91242bdb24a87 [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde])
I20251214 20:52:17.342085 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cafed2861e94cd9b0d10d33864ba45d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.342218 1202 tablet_bootstrap.cc:492] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.342401 1202 ts_tablet_manager.cc:1403] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.005s
W20251214 20:52:17.342890 851 leader_election.cc:343] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.342967 1661 tablet_loader.cc:96] loaded metadata for tablet 65e79db74fa046698507c83416ee1669 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e])
I20251214 20:52:17.343647 1661 tablet_loader.cc:96] loaded metadata for tablet 69f6108792fa4c2a9744e3248c76251f (table default.Table_b3f5ab40d9334de1a194aab3347387d7 [id=b470f1b24a6d4e7ab4420ae212e54ff0])
I20251214 20:52:17.343963 1661 tablet_loader.cc:96] loaded metadata for tablet 6a000951f61948ac85fdd431f3d162c2 (table default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904])
I20251214 20:52:17.344813 1661 tablet_loader.cc:96] loaded metadata for tablet 6c1893b3761c49ceb7cd71b3bd9f2cb4 (table default.Table_250b3222a83c44b08856444499fa612d [id=1ca50be529bd48398643cb6882958f77])
I20251214 20:52:17.345587 1661 tablet_loader.cc:96] loaded metadata for tablet 6f0a474ebca1401cb0fc7832e4602d6c (table default.Table_a40d3fbe67ba492d96c7db49c759836e [id=5308111e52e6484fbf050a603a18a3d7])
I20251214 20:52:17.346043 1661 tablet_loader.cc:96] loaded metadata for tablet 6fef1d6add764fd28a8f4995b337ea71 (table default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980])
I20251214 20:52:17.346782 1202 raft_consensus.cc:359] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.347061 1202 raft_consensus.cc:385] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.347191 1202 raft_consensus.cc:740] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.347474 1202 consensus_queue.cc:260] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.348342 1202 ts_tablet_manager.cc:1434] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:17.348816 1661 tablet_loader.cc:96] loaded metadata for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c])
I20251214 20:52:17.349299 1202 tablet_bootstrap.cc:492] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.350523 1661 tablet_loader.cc:96] loaded metadata for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_b92e50faaffb45c0a3e1c7a94166f985 [id=406157031d694ada8a9fcba4e4e23e90])
I20251214 20:52:17.352210 1661 tablet_loader.cc:96] loaded metadata for tablet 762068d560ff47918dd9fd8791083433 (table default.Table_431bf74dfb32400bb0b2ef098c18e8df [id=2fcb81bfc6904fdd924de2494aba2e4a])
I20251214 20:52:17.352905 1661 tablet_loader.cc:96] loaded metadata for tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9])
I20251214 20:52:17.353363 1661 tablet_loader.cc:96] loaded metadata for tablet 7dd657f3baff40bb8cb7f4648367f7a1 (table default.Table_8b86c42f7826440982c88a8625559146 [id=c35009dabd4a42598506e52b3d3e493c])
I20251214 20:52:17.356066 1661 tablet_loader.cc:96] loaded metadata for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (table default.Table_c845c10eeec54a7aa19cb8e0cebeb3a6 [id=4329d01e7f6147d0adad53fa86369b17])
I20251214 20:52:17.356492 1661 tablet_loader.cc:96] loaded metadata for tablet 88b9ca42a5ce4572aebe2a24f4758b12 (table default.Table_5242f3b282284ab7aff8525106732d53 [id=0d173bfba2b343dbac6a02a5ff84c828])
I20251214 20:52:17.356591 1202 tablet_bootstrap.cc:654] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.360213 1664 raft_consensus.cc:515] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.360373 1661 tablet_loader.cc:96] loaded metadata for tablet 8d72f55d2d2440d993aa205d96c356f1 (table default.Table_2f947b9f382942e082ea4909657a8f34 [id=66d124c2a8dd4cc188a832eff1dbe9ed])
I20251214 20:52:17.361029 1661 tablet_loader.cc:96] loaded metadata for tablet 8d74d3abc45a41969289c0a0892bfa04 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0])
I20251214 20:52:17.361564 1664 leader_election.cc:290] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.361848 1203 tablet_bootstrap.cc:654] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.362257 1661 tablet_loader.cc:96] loaded metadata for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be])
I20251214 20:52:17.362397 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd88d1e95ee04ef596863e1ae3687de0" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:17.362682 915 raft_consensus.cc:3060] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.362928 1661 tablet_loader.cc:96] loaded metadata for tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7])
I20251214 20:52:17.363358 1661 tablet_loader.cc:96] loaded metadata for tablet 96dc4bdabb234c3fa4c26bd1633e783d (table default.Table_5e8a4451754247c2add9a3245b63b050 [id=d7f40373ddad4286a8fe7966df2d9668])
I20251214 20:52:17.341442 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd88d1e95ee04ef596863e1ae3687de0" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.363693 1661 tablet_loader.cc:96] loaded metadata for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_e532e5fa17f2476a90df03eb6449a067 [id=ece0780edb0f41b787b46b290c4c0515])
I20251214 20:52:17.344146 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cafed2861e94cd9b0d10d33864ba45d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.346483 1717 raft_consensus.cc:493] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.364116 1661 tablet_loader.cc:96] loaded metadata for tablet 98b7cac3930a4a9eb16ac413c8f1e092 (table default.Table_e5f3db562155448e91e2e4314254cbe6 [id=826a2666d167435c8fdfd9ab8b8819e6])
I20251214 20:52:17.364100 1717 raft_consensus.cc:515] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.364483 1661 tablet_loader.cc:96] loaded metadata for tablet 9b1a937150ce4ecfa210c8465d682496 (table default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34])
I20251214 20:52:17.364979 1717 leader_election.cc:290] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.363090 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd88d1e95ee04ef596863e1ae3687de0" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.365284 1661 tablet_loader.cc:96] loaded metadata for tablet 9cfd9b2f9756433e9e52643417c4cd45 (table default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d])
W20251214 20:52:17.365469 989 leader_election.cc:343] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.365727 1661 tablet_loader.cc:96] loaded metadata for tablet 9d89ee602c524c96a21c0a235af07481 (table default.Table_bb6020a9d77b42b5955933a449edc238 [id=8872bfac95c64cbba1ccb84d1d9f2483])
I20251214 20:52:17.365949 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0279992d1f2419ab1caae116a0391b2" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.366171 1661 tablet_loader.cc:96] loaded metadata for tablet 9da5e98c6f4843449886176e9fcfa8e1 (table default.Table_b0d8d299bd0b4950948be859b03a3731 [id=5ded9fe825ae4308b5bd4b21c16634c8])
I20251214 20:52:17.366550 1661 tablet_loader.cc:96] loaded metadata for tablet 9e1835374cfa48939d72476870accc3b (table default.Table_dd97baf658ea4f44bba79f081b5ff75e [id=ba545f2c5ba949709f18c486d993faab])
I20251214 20:52:17.367697 1661 tablet_loader.cc:96] loaded metadata for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_6a0720013c0d470e91997a7ad8d483fd [id=88d5807cc76646e3a2962b5d36413894])
I20251214 20:52:17.368371 1661 tablet_loader.cc:96] loaded metadata for tablet a142157acbcd44adb6099b4428103ee5 (table default.Table_0870823e1f29409f9d61ebee0bbd060d [id=0a848dd5f6474712b2eba70add3743dc])
I20251214 20:52:17.369001 1661 tablet_loader.cc:96] loaded metadata for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf (table default.Table_08488285c9094a9dacc1e489c50271b3 [id=d3b4eef101ec491485c1889f08de9992])
I20251214 20:52:17.370059 1661 tablet_loader.cc:96] loaded metadata for tablet ac64f4fcb6a943668584658df19f950c (table default.Table_5275181d46dc4a85b27fc0c45fa587fe [id=ab9838348eb24201bcb2866b16994965])
W20251214 20:52:17.370311 989 leader_election.cc:343] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.370515 1661 tablet_loader.cc:96] loaded metadata for tablet afd51f330eb649b998a22a343d993b30 (table default.Table_cae5b2188af54d068db115a0745ee866 [id=f4157d7b9fc848a8b7fa08742b456167])
I20251214 20:52:17.372982 1661 tablet_loader.cc:96] loaded metadata for tablet b1f4ea72c6274569a98cab6afdc55d55 (table default.Table_893229542bec44e8b903bb0934bd16f5 [id=83b21f4baa574636ae3b50358fefae95])
I20251214 20:52:17.373426 1661 tablet_loader.cc:96] loaded metadata for tablet b2e83bca8ed642b6a1b705ef70fde27f (table default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7])
I20251214 20:52:17.373574 1735 raft_consensus.cc:493] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.374338 1661 tablet_loader.cc:96] loaded metadata for tablet b69bb4d5979f492683fc13238a668b9f (table default.Table_0dcbd217695546df9d1ca32583980907 [id=487c7f5559f64d13ae2de3da7c3e49a9])
I20251214 20:52:17.374810 1661 tablet_loader.cc:96] loaded metadata for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41])
I20251214 20:52:17.375777 1661 tablet_loader.cc:96] loaded metadata for tablet ba7952290e2c4746802cffeb14971da0 (table default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741])
W20251214 20:52:17.377071 848 leader_election.cc:343] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.376863 1735 raft_consensus.cc:515] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.378037 1735 leader_election.cc:290] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.378119 1661 tablet_loader.cc:96] loaded metadata for tablet bc73d590531c409e9f60670435dc8223 (table default.Table_59405cb9326245918d9cf6fb65ddbf40 [id=619272205b724c56ba3d2542c2c5879a])
I20251214 20:52:17.378392 848 leader_election.cc:304] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.378623 1661 tablet_loader.cc:96] loaded metadata for tablet bd8226b431c6456885e928880938969d (table default.Table_1917701c01bf48c7b2bfc335d05f6db0 [id=6fba2831971a4bffac786611b54a6468])
I20251214 20:52:17.379338 1661 tablet_loader.cc:96] loaded metadata for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9])
I20251214 20:52:17.379513 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9523d72fd91445539d67e1eac2a80afd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:17.380075 848 leader_election.cc:343] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.380534 1735 raft_consensus.cc:2749] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.381330 915 raft_consensus.cc:2468] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.382720 991 leader_election.cc:304] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:17.383090 1661 tablet_loader.cc:96] loaded metadata for tablet c6a8863ace4541aea7463af24db27142 (table default.Table_88881a27865d4b0aa6db6a2b736f0c17 [id=e0d1675617c74aa8a8afd81bd90f9639])
I20251214 20:52:17.383358 1667 raft_consensus.cc:2804] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.383522 1667 raft_consensus.cc:697] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:17.383606 1661 tablet_loader.cc:96] loaded metadata for tablet c9b7149a436a4cab80b39adccf8567f9 (table default.Table_67ec8d0cd3f948218b4637de6d68a57b [id=bdc2d86a977d4146a1a86717c5916807])
I20251214 20:52:17.383654 1735 consensus_queue.cc:1048] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.383810 1667 consensus_queue.cc:237] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.384050 1661 tablet_loader.cc:96] loaded metadata for tablet ca263a80702f4f64aa7c872c2be08207 (table default.Table_642da83f7ed0444688afdf91b7a9e7e8 [id=bca3a2d37f8b4fc797a5c52a43c14869])
I20251214 20:52:17.384599 1206 tablet_bootstrap.cc:492] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:17.384716 1661 tablet_loader.cc:96] loaded metadata for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956])
I20251214 20:52:17.384790 1206 ts_tablet_manager.cc:1403] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.002s
I20251214 20:52:17.385113 1661 tablet_loader.cc:96] loaded metadata for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05])
I20251214 20:52:17.386065 1661 tablet_loader.cc:96] loaded metadata for tablet cf04f91e79754569a51d468ab576b0dc (table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d])
I20251214 20:52:17.386088 1206 raft_consensus.cc:359] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.386323 1206 raft_consensus.cc:385] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.386402 1206 raft_consensus.cc:740] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.386583 1661 tablet_loader.cc:96] loaded metadata for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc])
I20251214 20:52:17.386656 1206 consensus_queue.cc:260] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.387082 1661 tablet_loader.cc:96] loaded metadata for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10])
I20251214 20:52:17.387413 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9523d72fd91445539d67e1eac2a80afd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.388797 1661 tablet_loader.cc:96] loaded metadata for tablet d424e3387698434f92e84eb1d43f5949 (table default.Table_ee7f67ff1a1c4014a1cc4f9cf7f6613b [id=1597e24e3fff4007ab66b44505c960d8])
I20251214 20:52:17.389937 1661 tablet_loader.cc:96] loaded metadata for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_3736aabecd40417f9cb502ac612bacee [id=fc592ea105f242a299337c9cbe438778])
I20251214 20:52:17.390694 1206 ts_tablet_manager.cc:1434] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:17.391392 1206 tablet_bootstrap.cc:492] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:17.393493 1661 tablet_loader.cc:96] loaded metadata for tablet e16fe751f10f4616a00d0af811998729 (table default.Table_181aab73cbf1461b85364fc893700db7 [id=8fd9be1206a74aa1921e0dade8be3774])
I20251214 20:52:17.393966 1661 tablet_loader.cc:96] loaded metadata for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_2ae9b9f412384e96bfa1e3e6d26ea104 [id=9553b46856c94003a293135c814a7df7])
I20251214 20:52:17.394345 1661 tablet_loader.cc:96] loaded metadata for tablet e6ebfc373fbf41b1920c271a15302484 (table default.Table_26795454cb054bd6bdb4100765bd3c17 [id=1a3349421ffd4a7d933c8301fffb9154])
I20251214 20:52:17.394754 1661 tablet_loader.cc:96] loaded metadata for tablet e8c02a2e4a2640b1b3a69f25a708684b (table default.Table_a519bf6d1ab8419784be3772e899869d [id=5cbe2438b4294442a9778bf1f641b60e])
I20251214 20:52:17.396332 1661 tablet_loader.cc:96] loaded metadata for tablet e90d11d69b8d4fd49150aa67a8190e5b (table default.Table_cf7fd4bd0b764cc79acc374ea7bb5187 [id=b3df2580f4a74189aa06f021d6b4ce71])
I20251214 20:52:17.396968 1661 tablet_loader.cc:96] loaded metadata for tablet eb29fd8c85344ed8aa8d880e2ed240bc (table default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 [id=23efc0ee36244ca0a6ff0f2faeb00b1f])
I20251214 20:52:17.397420 1661 tablet_loader.cc:96] loaded metadata for tablet ebb26a5ae7224e9ca8a513c7b5013618 (table default.Table_4907bb640d114586b98e6fa0e514cb5a [id=2a6d9d4658f24eb5b39adea101a0891c])
I20251214 20:52:17.399010 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0279992d1f2419ab1caae116a0391b2" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.399484 915 raft_consensus.cc:2393] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:17.400811 1661 tablet_loader.cc:96] loaded metadata for tablet f900ebe92efa476cac232131a818d449 (table default.Table_5de120dfaab44b84868db0d1f79e2cee [id=1158f8a25d2b4696a5f5ed2c3a426862])
I20251214 20:52:17.401393 1661 tablet_loader.cc:96] loaded metadata for tablet f945814952b14131ba4e0c67a403a680 (table default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4])
I20251214 20:52:17.401661 712 leader_election.cc:304] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.401777 1661 tablet_loader.cc:96] loaded metadata for tablet fc7353593bda4fceaa389c393ddc19c6 (table default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589])
I20251214 20:52:17.402043 1717 raft_consensus.cc:3060] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.402244 1661 tablet_loader.cc:96] loaded metadata for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 (table default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8])
I20251214 20:52:17.402555 1661 tablet_loader.cc:96] loaded metadata for tablet fe24e4f4e74040249068505058c15622 (table default.Table_00bef63ea36c4a7585c2832be1454b68 [id=6a279e57008549c593df838211cc9cc1])
W20251214 20:52:17.402798 712 consensus_peers.cc:597] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1df0dad2fb040a5a12e0401e00c87db. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:17.402882 1661 tablet_loader.cc:96] loaded metadata for tablet fe4086fc7a4f4db0b7541f06c9833aab (table default.Table_8076bc8b31ba411faf089f314e6ed2ef [id=25016f04bb864f92a1cdfb6ceaa9fba9])
I20251214 20:52:17.403424 1653 raft_consensus.cc:493] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.403578 1653 raft_consensus.cc:515] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.404369 1653 leader_election.cc:290] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.403249 1661 tablet_loader.cc:96] loaded metadata for tablet feb19ea389af4ea6bb158a0e8aaeb094 (table default.Table_55706259657e4a09aa794297547e9d98 [id=59d9716c5cee42c391018111323c5181])
W20251214 20:52:17.403975 851 leader_election.cc:343] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.405004 1717 raft_consensus.cc:2749] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.405215 1661 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 20:52:17.405607 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a10663bd306d40e7a6375d06d69be22d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.405973 851 leader_election.cc:304] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.406276 1653 raft_consensus.cc:2749] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:17.406689 848 leader_election.cc:343] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.407680 1203 tablet_bootstrap.cc:492] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.407892 1203 ts_tablet_manager.cc:1403] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.000s
I20251214 20:52:17.408517 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a10663bd306d40e7a6375d06d69be22d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.408866 1203 raft_consensus.cc:359] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.409109 1203 raft_consensus.cc:385] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.409250 1203 raft_consensus.cc:740] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:17.409545 851 leader_election.cc:343] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.409576 1203 consensus_queue.cc:260] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.410410 1203 ts_tablet_manager.cc:1434] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:17.410745 1203 tablet_bootstrap.cc:492] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.411113 1202 tablet_bootstrap.cc:492] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.411392 1202 ts_tablet_manager.cc:1403] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.007s
I20251214 20:52:17.411078 851 leader_election.cc:304] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.412464 1653 raft_consensus.cc:2749] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.412490 1202 raft_consensus.cc:359] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.412808 1202 raft_consensus.cc:385] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.412914 1202 raft_consensus.cc:740] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.413218 1202 consensus_queue.cc:260] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.413856 1202 ts_tablet_manager.cc:1434] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:17.416118 1202 tablet_bootstrap.cc:492] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.417874 1203 tablet_bootstrap.cc:654] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.418119 1202 tablet_bootstrap.cc:654] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.422420 1661 catalog_manager.cc:1269] Loaded cluster ID: c3f04138608d4adc935bb66e1a72f2bb
I20251214 20:52:17.422592 1661 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 20:52:17.423242 1206 tablet_bootstrap.cc:654] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:17.425957 989 consensus_peers.cc:597] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84a0237e4ee34f94821d8647a400575b. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:17.428503 991 consensus_peers.cc:597] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84a0237e4ee34f94821d8647a400575b. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:17.430850 1206 tablet_bootstrap.cc:492] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:17.431603 1661 catalog_manager.cc:1514] Loading token signing keys...
I20251214 20:52:17.434803 1206 ts_tablet_manager.cc:1403] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.002s
I20251214 20:52:17.435961 1206 raft_consensus.cc:359] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.437141 1206 raft_consensus.cc:385] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.439953 1661 catalog_manager.cc:6038] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Loaded TSK: 0
I20251214 20:52:17.440959 1661 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 20:52:17.442991 1206 raft_consensus.cc:740] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.444259 1203 tablet_bootstrap.cc:492] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.444485 1203 ts_tablet_manager.cc:1403] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.001s
I20251214 20:52:17.445377 1203 raft_consensus.cc:359] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.445600 1203 raft_consensus.cc:385] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.445690 1203 raft_consensus.cc:740] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.445991 1203 consensus_queue.cc:260] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.446660 1203 ts_tablet_manager.cc:1434] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:17.447588 1203 tablet_bootstrap.cc:492] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.449371 1206 consensus_queue.cc:260] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.450186 1206 ts_tablet_manager.cc:1434] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251214 20:52:17.449572 1664 consensus_queue.cc:1048] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.452373 1202 tablet_bootstrap.cc:492] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.452581 1202 ts_tablet_manager.cc:1403] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.001s
I20251214 20:52:17.452735 1203 tablet_bootstrap.cc:654] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.453559 1202 raft_consensus.cc:359] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.453794 1202 raft_consensus.cc:385] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.453868 1202 raft_consensus.cc:740] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.454084 1206 tablet_bootstrap.cc:492] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:17.454119 1202 consensus_queue.cc:260] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.454635 1202 ts_tablet_manager.cc:1434] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:17.454926 1202 tablet_bootstrap.cc:492] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.455683 1735 raft_consensus.cc:493] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.455847 1735 raft_consensus.cc:515] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.456552 1735 leader_election.cc:290] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.458668 1202 tablet_bootstrap.cc:654] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.458371 455 catalog_manager.cc:5654] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } }
I20251214 20:52:17.459455 455 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3fa0317e999b483788e5c5652e6e83de (table default.Table_b01c45a1fe8f4a71858fcb2127177471 [id=ef22180b36a241fea13d0e622c02b50a]). Expected version 1 got 0
I20251214 20:52:17.462471 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8af99e0bd8774a678575847a9ec2e557" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.463187 1717 raft_consensus.cc:493] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.465858 1735 raft_consensus.cc:493] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.466065 1735 raft_consensus.cc:515] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.466732 1735 leader_election.cc:290] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.468389 1203 tablet_bootstrap.cc:492] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.468612 1203 ts_tablet_manager.cc:1403] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251214 20:52:17.469578 1203 raft_consensus.cc:359] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.469797 1203 raft_consensus.cc:385] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.469905 1203 raft_consensus.cc:740] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.470247 1203 consensus_queue.cc:260] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.470913 1203 ts_tablet_manager.cc:1434] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:17.471388 1203 tablet_bootstrap.cc:492] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.473640 1203 tablet_bootstrap.cc:654] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.474103 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe4086fc7a4f4db0b7541f06c9833aab" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:17.476081 851 leader_election.cc:343] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:17.476826 851 leader_election.cc:343] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.479300 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8af99e0bd8774a678575847a9ec2e557" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:17.480108 848 leader_election.cc:343] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.480274 848 leader_election.cc:304] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.481295 1735 raft_consensus.cc:2749] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.481484 1735 raft_consensus.cc:493] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.481621 1735 raft_consensus.cc:515] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.482295 1735 leader_election.cc:290] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.483865 452 master_service.cc:954] ReplaceTablet: received request to replace tablet b2e83bca8ed642b6a1b705ef70fde27f from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:17.483966 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b151b87a38944a7a89d0fe6ff10ae68b" } modify_external_catalogs: true
I20251214 20:52:17.484329 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b151b87a38944a7a89d0fe6ff10ae68b" } modify_external_catalogs: true
I20251214 20:52:17.484855 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_1917701c01bf48c7b2bfc335d05f6db0" } modify_external_catalogs: true
I20251214 20:52:17.485190 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_1917701c01bf48c7b2bfc335d05f6db0" } modify_external_catalogs: true
I20251214 20:52:17.487193 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_67ec8d0cd3f948218b4637de6d68a57b" } modify_external_catalogs: true
I20251214 20:52:17.487426 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_67ec8d0cd3f948218b4637de6d68a57b" } modify_external_catalogs: true
I20251214 20:52:17.492390 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a40d3fbe67ba492d96c7db49c759836e" } modify_external_catalogs: true
I20251214 20:52:17.492651 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a40d3fbe67ba492d96c7db49c759836e" } modify_external_catalogs: true
I20251214 20:52:17.493685 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c16895ca08d94b56b5c4bc9352e330ef" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.493738 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_e532e5fa17f2476a90df03eb6449a067" } new_table_name: "default.Table_cd43be14c9f84889bd6e03167dd35f6c" modify_external_catalogs: true
I20251214 20:52:17.493945 1056 raft_consensus.cc:2468] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:17.494598 851 leader_election.cc:304] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.463423 1717 raft_consensus.cc:515] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.497247 1735 raft_consensus.cc:2804] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.497380 1735 raft_consensus.cc:493] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.497488 1735 raft_consensus.cc:3060] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.501788 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c16895ca08d94b56b5c4bc9352e330ef" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:17.502976 848 leader_election.cc:343] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.506881 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2842f1dc0c374d24946f90dac1cf8aa5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.506958 1717 leader_election.cc:290] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.507190 911 raft_consensus.cc:2468] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:17.501052 449 master_service.cc:954] ReplaceTablet: received request to replace tablet 3fd0fa0dad974c98bf6f52c8cfe83035 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:17.507637 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe4086fc7a4f4db0b7541f06c9833aab" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.508101 712 leader_election.cc:304] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.508512 1717 raft_consensus.cc:2804] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.508074 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2842f1dc0c374d24946f90dac1cf8aa5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.508976 1717 raft_consensus.cc:493] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.509091 1717 raft_consensus.cc:3060] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:17.509701 848 leader_election.cc:343] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.509855 848 leader_election.cc:304] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.511951 1717 raft_consensus.cc:515] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.513455 1206 tablet_bootstrap.cc:654] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.513587 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2842f1dc0c374d24946f90dac1cf8aa5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:17.514112 1664 raft_consensus.cc:493] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.514348 1664 raft_consensus.cc:515] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.514973 1717 leader_election.cc:290] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.515152 1664 leader_election.cc:290] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.515236 1056 raft_consensus.cc:2468] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:17.516772 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b652a8a80246443491551e76e68d2a8b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.517058 776 raft_consensus.cc:2468] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.517635 989 leader_election.cc:304] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:17.518049 1664 raft_consensus.cc:2804] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.518173 1664 raft_consensus.cc:493] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.518371 1664 raft_consensus.cc:3060] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.521633 1664 raft_consensus.cc:515] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.525245 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b652a8a80246443491551e76e68d2a8b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.525537 776 raft_consensus.cc:3060] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.527614 1203 tablet_bootstrap.cc:492] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.527817 1203 ts_tablet_manager.cc:1403] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.006s
I20251214 20:52:17.502015 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_0dbf8224a9254c70a34ecf1cbcc79306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:17.528767 776 raft_consensus.cc:2468] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.528785 1203 raft_consensus.cc:359] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.529074 1203 raft_consensus.cc:385] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.529155 1203 raft_consensus.cc:740] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.529398 1203 consensus_queue.cc:260] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:17.529567 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dbf8224a9254c70a34ecf1cbcc79306 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:17.530195 989 leader_election.cc:304] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:17.531157 1664 leader_election.cc:290] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.531407 1664 raft_consensus.cc:2804] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.531658 1664 raft_consensus.cc:697] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:17.532032 1664 consensus_queue.cc:237] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.533263 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 012b81223d0f4ae9b742014e329f11a9
I20251214 20:52:17.534806 1203 ts_tablet_manager.cc:1434] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:17.535300 1203 tablet_bootstrap.cc:492] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.537209 1203 tablet_bootstrap.cc:654] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.545352 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
/tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:34819
--webserver_interface=127.0.104.61
--webserver_port=34369
--builtin_ntp_servers=127.0.104.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251214 20:52:17.549463 1203 tablet_bootstrap.cc:492] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.549643 1203 ts_tablet_manager.cc:1403] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251214 20:52:17.552349 1203 raft_consensus.cc:359] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.552577 1203 raft_consensus.cc:385] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.552682 1203 raft_consensus.cc:740] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.553195 1667 raft_consensus.cc:493] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.553355 1667 raft_consensus.cc:515] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.554023 1667 leader_election.cc:290] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.552990 1203 consensus_queue.cc:260] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:17.556190 991 leader_election.cc:343] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 63497e77baeb4d328be2421b5732a28a
I20251214 20:52:17.556711 1203 ts_tablet_manager.cc:1434] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251214 20:52:17.557055 1203 tablet_bootstrap.cc:492] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.562841 1102 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:17.571507 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63497e77baeb4d328be2421b5732a28a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.571851 776 raft_consensus.cc:2468] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.572422 989 leader_election.cc:304] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.573042 1667 raft_consensus.cc:2804] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.573182 1667 raft_consensus.cc:493] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.573393 1667 raft_consensus.cc:3060] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.576563 1667 raft_consensus.cc:515] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.510308 1650 consensus_queue.cc:1048] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.067s
I20251214 20:52:17.510399 1653 raft_consensus.cc:2749] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.577433 1650 raft_consensus.cc:493] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.577647 1650 raft_consensus.cc:515] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.577968 1667 leader_election.cc:290] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.516177 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b652a8a80246443491551e76e68d2a8b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.573194 1203 tablet_bootstrap.cc:654] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.578231 911 raft_consensus.cc:2468] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.578387 1650 leader_election.cc:290] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.578775 1735 raft_consensus.cc:515] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.579649 1735 leader_election.cc:290] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.579989 1202 tablet_bootstrap.cc:492] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.580148 1202 ts_tablet_manager.cc:1403] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.125s user 0.001s sys 0.012s
I20251214 20:52:17.516338 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2842f1dc0c374d24946f90dac1cf8aa5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:17.580709 914 raft_consensus.cc:3060] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.581053 1202 raft_consensus.cc:359] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.581251 1202 raft_consensus.cc:385] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.581378 1202 raft_consensus.cc:740] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.581619 1202 consensus_queue.cc:260] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.582382 1202 ts_tablet_manager.cc:1434] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:17.582796 1202 tablet_bootstrap.cc:492] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.524454 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b652a8a80246443491551e76e68d2a8b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:17.583393 913 raft_consensus.cc:3060] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.585424 1202 tablet_bootstrap.cc:654] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.592469 1206 tablet_bootstrap.cc:492] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:17.592775 1206 ts_tablet_manager.cc:1403] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.001s
I20251214 20:52:17.593742 1206 raft_consensus.cc:359] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.594054 1206 raft_consensus.cc:385] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.594166 1206 raft_consensus.cc:740] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.594527 1206 consensus_queue.cc:260] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.596195 1202 tablet_bootstrap.cc:492] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.596474 1202 ts_tablet_manager.cc:1403] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20251214 20:52:17.596895 1206 ts_tablet_manager.cc:1434] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251214 20:52:17.597271 1206 tablet_bootstrap.cc:492] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:17.597357 1202 raft_consensus.cc:359] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.597615 1202 raft_consensus.cc:385] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.597711 1202 raft_consensus.cc:740] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.597966 1202 consensus_queue.cc:260] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.598542 1202 ts_tablet_manager.cc:1434] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251214 20:52:17.599009 991 leader_election.cc:343] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 63497e77baeb4d328be2421b5732a28a
I20251214 20:52:17.598850 1760 raft_consensus.cc:493] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.599285 1760 raft_consensus.cc:515] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.600206 1760 leader_election.cc:290] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.600548 1202 tablet_bootstrap.cc:492] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.601661 913 raft_consensus.cc:2468] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.602762 1202 tablet_bootstrap.cc:654] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.608680 1667 raft_consensus.cc:493] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.608893 1667 raft_consensus.cc:515] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.609645 1667 leader_election.cc:290] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.610931 1202 tablet_bootstrap.cc:492] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.611726 1202 ts_tablet_manager.cc:1403] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20251214 20:52:17.613454 1202 raft_consensus.cc:359] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.614326 1202 raft_consensus.cc:385] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.614558 1202 raft_consensus.cc:740] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.614957 1202 consensus_queue.cc:260] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.615934 1202 ts_tablet_manager.cc:1434] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:17.617969 1203 tablet_bootstrap.cc:492] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.618199 1203 ts_tablet_manager.cc:1403] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.010s
I20251214 20:52:17.618271 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd51f330eb649b998a22a343d993b30" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.618952 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c16895ca08d94b56b5c4bc9352e330ef" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:17.619050 1203 raft_consensus.cc:359] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.619242 1056 raft_consensus.cc:3060] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.619335 1203 raft_consensus.cc:385] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.619486 1203 raft_consensus.cc:740] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.619781 1203 consensus_queue.cc:260] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.620376 1203 ts_tablet_manager.cc:1434] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:17.620731 1203 tablet_bootstrap.cc:492] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.621455 1667 raft_consensus.cc:493] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.621628 1667 raft_consensus.cc:515] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.622190 1056 raft_consensus.cc:2468] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:17.622321 1667 leader_election.cc:290] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.622803 1203 tablet_bootstrap.cc:654] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.623114 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b86f4adaf4540f48a4f42bc45e259e5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.627058 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 012b81223d0f4ae9b742014e329f11a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:17.629035 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63497e77baeb4d328be2421b5732a28a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:17.629125 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 012b81223d0f4ae9b742014e329f11a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012b81223d0f4ae9b742014e329f11a9 already in progress: creating tablet
I20251214 20:52:17.629273 776 raft_consensus.cc:3060] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.629632 1203 tablet_bootstrap.cc:492] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.629631 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9523d72fd91445539d67e1eac2a80afd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.629796 1203 ts_tablet_manager.cc:1403] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20251214 20:52:17.630023 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b4f582d8165439d86d9228c146dc41a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.630223 773 raft_consensus.cc:2468] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.630759 1717 raft_consensus.cc:493] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.630699 1203 raft_consensus.cc:359] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.630878 1203 raft_consensus.cc:385] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.630965 1203 raft_consensus.cc:740] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.630905 1717 raft_consensus.cc:515] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.631191 1203 consensus_queue.cc:260] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.631755 1203 ts_tablet_manager.cc:1434] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:17.631816 1717 leader_election.cc:290] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:17.632131 1203 tablet_bootstrap.cc:492] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.633106 776 raft_consensus.cc:2468] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.634253 1203 tablet_bootstrap.cc:654] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.635788 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6bac3fed78714367b43db88be8ee1712" } new_table_name: "default.Table_31645b7a21d345ef9bc06c0ba9b2c5fa" modify_external_catalogs: true
I20251214 20:52:17.636933 1203 tablet_bootstrap.cc:492] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.637130 1203 ts_tablet_manager.cc:1403] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20251214 20:52:17.638010 1203 raft_consensus.cc:359] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.638262 1203 raft_consensus.cc:385] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.638360 1203 raft_consensus.cc:740] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.638638 1203 consensus_queue.cc:260] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.639261 1203 ts_tablet_manager.cc:1434] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:17.639601 1203 tablet_bootstrap.cc:492] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.640174 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ad73c92d80cf4c5d9f102d239903cb19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:17.640897 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad73c92d80cf4c5d9f102d239903cb19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:17.641575 1203 tablet_bootstrap.cc:654] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.642516 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6a0720013c0d470e91997a7ad8d483fd" } new_table_name: "default.Table_e9b7a7c58ab24328adb161b3cdd1c356" modify_external_catalogs: true
I20251214 20:52:17.643808 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_488f7bbd18364d1e99b1f57aa7aba2c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:17.644335 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_488f7bbd18364d1e99b1f57aa7aba2c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:17.644467 823 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:17.649058 453 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 6f0a474ebca1401cb0fc7832e4602d6c
I20251214 20:52:17.649998 1203 tablet_bootstrap.cc:492] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.650166 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet bd8226b431c6456885e928880938969d
I20251214 20:52:17.650177 1203 ts_tablet_manager.cc:1403] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20251214 20:52:17.651049 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c3fc0fa4ec254ff889443791279bb8af" } new_table_name: "default.Table_b4334a1d99a54c2eabe2fb909563313c" modify_external_catalogs: true
I20251214 20:52:17.651158 1203 raft_consensus.cc:359] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.651351 1203 raft_consensus.cc:385] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.651445 1203 raft_consensus.cc:740] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.651667 1203 consensus_queue.cc:260] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.652345 1203 ts_tablet_manager.cc:1434] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:17.652657 1203 tablet_bootstrap.cc:492] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.654721 1203 tablet_bootstrap.cc:654] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.658468 1203 tablet_bootstrap.cc:492] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.658694 1203 ts_tablet_manager.cc:1403] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
W20251214 20:52:17.659363 989 leader_election.cc:343] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.659669 1203 raft_consensus.cc:359] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.659932 1203 raft_consensus.cc:385] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.660043 1203 raft_consensus.cc:740] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.660380 1203 consensus_queue.cc:260] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.660710 989 leader_election.cc:304] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:17.660959 1203 ts_tablet_manager.cc:1434] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:17.661154 1667 raft_consensus.cc:2804] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.661255 1667 raft_consensus.cc:493] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.661303 1203 tablet_bootstrap.cc:492] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.661348 1667 raft_consensus.cc:3060] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.663189 1203 tablet_bootstrap.cc:654] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.664129 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet bd8226b431c6456885e928880938969d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:17.664309 1667 raft_consensus.cc:515] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.664530 1774 tablet_replica.cc:333] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:17.664757 1774 raft_consensus.cc:2243] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:17.665056 1774 raft_consensus.cc:2272] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:17.665479 1203 tablet_bootstrap.cc:492] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.665613 1203 ts_tablet_manager.cc:1403] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20251214 20:52:17.665915 1667 leader_election.cc:290] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.666317 1774 ts_tablet_manager.cc:1916] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:17.666548 1203 raft_consensus.cc:359] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.666757 1203 raft_consensus.cc:385] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.666994 1203 raft_consensus.cc:740] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.667316 1203 consensus_queue.cc:260] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.668061 1203 ts_tablet_manager.cc:1434] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:17.668426 1203 tablet_bootstrap.cc:492] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.670424 1203 tablet_bootstrap.cc:654] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.673553 1203 tablet_bootstrap.cc:492] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:17.673767 1203 ts_tablet_manager.cc:1403] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20251214 20:52:17.673949 1774 ts_tablet_manager.cc:1929] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:17.674619 1203 raft_consensus.cc:359] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.675180 1774 log.cc:1199] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/bd8226b431c6456885e928880938969d
I20251214 20:52:17.675267 1203 raft_consensus.cc:385] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.675561 1203 raft_consensus.cc:740] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.675765 1202 tablet_bootstrap.cc:492] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:17.675877 1203 consensus_queue.cc:260] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.676558 1760 raft_consensus.cc:493] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.676460 1203 ts_tablet_manager.cc:1434] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:17.677057 1774 ts_tablet_manager.cc:1950] T bd8226b431c6456885e928880938969d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:17.677609 1206 tablet_bootstrap.cc:654] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.678460 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b86f4adaf4540f48a4f42bc45e259e5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.678877 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd51f330eb649b998a22a343d993b30" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.679311 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c16895ca08d94b56b5c4bc9352e330ef" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.677137 1203 tablet_bootstrap.cc:492] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:17.679474 914 raft_consensus.cc:2468] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:17.679792 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9523d72fd91445539d67e1eac2a80afd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.681157 1203 tablet_bootstrap.cc:654] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.682821 1664 consensus_queue.cc:1048] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.683308 1762 raft_consensus.cc:493] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:17.684778 991 leader_election.cc:343] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 6b4f582d8165439d86d9228c146dc41a
W20251214 20:52:17.685225 991 leader_election.cc:343] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 6b4f582d8165439d86d9228c146dc41a
I20251214 20:52:17.685582 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd88d1e95ee04ef596863e1ae3687de0" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.695068 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 012b81223d0f4ae9b742014e329f11a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:17.695277 910 raft_consensus.cc:2468] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.696164 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet bd8226b431c6456885e928880938969d (table default.Table_1917701c01bf48c7b2bfc335d05f6db0 [id=6fba2831971a4bffac786611b54a6468]) successfully deleted
I20251214 20:52:17.696390 991 leader_election.cc:304] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:17.697085 1664 raft_consensus.cc:2804] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.697233 1664 raft_consensus.cc:493] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.697381 1664 raft_consensus.cc:3060] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.701637 1664 raft_consensus.cc:515] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.703212 1664 leader_election.cc:290] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.703377 1206 tablet_bootstrap.cc:492] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:17.703668 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd88d1e95ee04ef596863e1ae3687de0" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.704470 1206 ts_tablet_manager.cc:1403] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.107s user 0.002s sys 0.009s
I20251214 20:52:17.705327 1206 raft_consensus.cc:359] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.705539 1206 raft_consensus.cc:385] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:17.705665 1206 raft_consensus.cc:740] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:17.705996 1206 consensus_queue.cc:260] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.706673 1206 ts_tablet_manager.cc:1434] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:17.707031 1206 tablet_bootstrap.cc:492] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:17.707989 1760 raft_consensus.cc:515] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.708842 1206 tablet_bootstrap.cc:654] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.708894 1760 leader_election.cc:290] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:17.709154 851 leader_election.cc:343] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.709450 851 leader_election.cc:304] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.711205 1760 raft_consensus.cc:2804] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.711370 1760 raft_consensus.cc:697] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:17.712440 913 raft_consensus.cc:2393] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:17.712265 1760 consensus_queue.cc:237] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.713056 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9523d72fd91445539d67e1eac2a80afd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:17.714900 910 raft_consensus.cc:3060] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.703989 1762 raft_consensus.cc:515] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.721545 910 raft_consensus.cc:2468] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.722786 991 leader_election.cc:304] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.724727 1202 tablet_bootstrap.cc:654] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:17.746273 1762 leader_election.cc:290] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.748596 1667 raft_consensus.cc:2804] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.748833 1667 raft_consensus.cc:697] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:17.749109 892 tablet_service.cc:1552] Processing DeleteTablet for tablet bd8226b431c6456885e928880938969d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:17.749173 1667 consensus_queue.cc:237] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
W20251214 20:52:17.749366 848 leader_election.cc:343] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:17.749565 851 leader_election.cc:343] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.750017 851 leader_election.cc:304] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.750360 1595 raft_consensus.cc:493] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.750530 1595 raft_consensus.cc:515] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.751175 1595 leader_election.cc:290] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:17.751662 712 leader_election.cc:304] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:17.752457 1717 raft_consensus.cc:2804] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.752645 1717 raft_consensus.cc:697] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:17.752957 1717 consensus_queue.cc:237] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:17.755720 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf04f91e79754569a51d468ab576b0dc" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.755975 1056 raft_consensus.cc:2468] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:17.756858 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2ae9dc8ead8403cb7215af8778872f0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.757091 1055 raft_consensus.cc:2468] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:17.758469 1762 raft_consensus.cc:2749] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.759366 851 leader_election.cc:304] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.759806 851 leader_election.cc:304] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.760228 1653 raft_consensus.cc:2804] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.760396 1653 raft_consensus.cc:493] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.760392 1762 raft_consensus.cc:2804] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.760527 1653 raft_consensus.cc:3060] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.760612 1762 raft_consensus.cc:493] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.760733 1762 raft_consensus.cc:3060] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.761816 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2ae9dc8ead8403cb7215af8778872f0" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.771437 1735 raft_consensus.cc:493] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.771667 1735 raft_consensus.cc:515] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.772432 1735 leader_election.cc:290] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:17.773483 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 92320.2us
I20251214 20:52:17.773561 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34eb8fa3e34427495fa5985b7ba81bf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:17.774207 851 leader_election.cc:343] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.781426 1653 raft_consensus.cc:515] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.782121 1653 leader_election.cc:290] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.786984 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf04f91e79754569a51d468ab576b0dc" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:17.787254 1055 raft_consensus.cc:3060] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.789525 1762 raft_consensus.cc:515] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.790259 1055 raft_consensus.cc:2468] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:17.790997 1762 leader_election.cc:290] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.792281 851 leader_election.cc:304] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.792785 1667 consensus_queue.cc:1048] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.793083 910 raft_consensus.cc:2393] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63848a9fb78a4013a6f1657621272d75 in current term 1: Already voted for candidate a2e8fd1cb731446d88f8e510c7a52999 in this term.
I20251214 20:52:17.793330 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2ae9dc8ead8403cb7215af8778872f0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:17.793859 1762 raft_consensus.cc:2804] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.794112 1762 raft_consensus.cc:697] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:17.794396 1055 raft_consensus.cc:3060] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.794700 1762 consensus_queue.cc:237] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.801827 1055 raft_consensus.cc:2468] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:17.802891 851 leader_election.cc:304] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.803372 1653 raft_consensus.cc:2804] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.803678 1653 raft_consensus.cc:697] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:17.804062 1653 consensus_queue.cc:237] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.811640 452 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet b2e83bca8ed642b6a1b705ef70fde27f
I20251214 20:52:17.812321 452 catalog_manager.cc:6724] ReplaceTablet: tablet b2e83bca8ed642b6a1b705ef70fde27f deleted and replaced by tablet f11834cabdad445b8e53cd237d146282
I20251214 20:52:17.813818 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f with delete_type TABLET_DATA_DELETED (replaced by tablet f11834cabdad445b8e53cd237d146282) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:17.814491 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b2e83bca8ed642b6a1b705ef70fde27f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2e83bca8ed642b6a1b705ef70fde27f already in progress: creating tablet
I20251214 20:52:17.814718 892 tablet_service.cc:1552] Processing DeleteTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f with delete_type TABLET_DATA_DELETED (replaced by tablet f11834cabdad445b8e53cd237d146282) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:17.773869 848 leader_election.cc:343] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.819113 848 leader_election.cc:304] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:17.819633 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 45754.3us
W20251214 20:52:17.820048 848 leader_election.cc:343] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.820817 1667 consensus_queue.cc:1048] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.821048 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf04f91e79754569a51d468ab576b0dc" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:17.822083 991 consensus_peers.cc:597] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 649f5bd67cbb4f61b7066269f0bf9266. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:17.823102 1653 raft_consensus.cc:2749] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.824860 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2ae9dc8ead8403cb7215af8778872f0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.825888 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34eb8fa3e34427495fa5985b7ba81bf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.826843 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf04f91e79754569a51d468ab576b0dc" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.827883 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2ae9dc8ead8403cb7215af8778872f0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.830420 989 leader_election.cc:304] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.830456 1717 consensus_queue.cc:1048] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.831257 713 leader_election.cc:304] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:17.831302 1667 consensus_queue.cc:1048] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.831597 1717 raft_consensus.cc:3060] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.831830 1667 raft_consensus.cc:2804] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.831951 1667 raft_consensus.cc:697] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:17.832221 1667 consensus_queue.cc:237] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.834512 1717 raft_consensus.cc:2749] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.835656 1762 consensus_queue.cc:1048] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.864534 1664 raft_consensus.cc:493] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.864833 1664 raft_consensus.cc:515] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.865777 1664 leader_election.cc:290] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:17.865947 1667 raft_consensus.cc:493] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.866153 1667 raft_consensus.cc:515] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.866710 1667 leader_election.cc:290] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.867327 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "472e4a5990e64d02b1d52dc6e86d7190" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.867574 911 raft_consensus.cc:2468] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.867498 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52daf1204bf142e7b82852bdec6582fb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.867812 913 raft_consensus.cc:2468] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.868057 991 leader_election.cc:304] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.868548 991 leader_election.cc:304] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.868999 1667 raft_consensus.cc:2804] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.869086 1667 raft_consensus.cc:493] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.869180 1667 raft_consensus.cc:3060] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.872014 1667 raft_consensus.cc:515] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.874825 1667 leader_election.cc:290] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.874838 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "472e4a5990e64d02b1d52dc6e86d7190" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:17.875203 913 raft_consensus.cc:3060] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.875342 1667 raft_consensus.cc:2804] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.875471 1667 raft_consensus.cc:493] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.875566 1667 raft_consensus.cc:3060] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.878237 1667 raft_consensus.cc:515] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.878479 913 raft_consensus.cc:2468] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.879046 991 leader_election.cc:304] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.883674 1667 leader_election.cc:290] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:17.883667 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52daf1204bf142e7b82852bdec6582fb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:17.884136 913 raft_consensus.cc:3060] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.884187 1667 raft_consensus.cc:2804] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.884732 1667 raft_consensus.cc:697] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:17.885315 1667 consensus_queue.cc:237] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:17.888309 913 raft_consensus.cc:2468] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.888549 1664 consensus_queue.cc:1048] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:17.890008 991 leader_election.cc:304] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.890339 1799 raft_consensus.cc:2804] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.890548 1799 raft_consensus.cc:697] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:17.892313 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b4f582d8165439d86d9228c146dc41a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.892957 776 raft_consensus.cc:3060] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.897135 776 raft_consensus.cc:2468] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.890900 1799 consensus_queue.cc:237] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.899235 1762 raft_consensus.cc:493] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.900091 1762 raft_consensus.cc:515] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.900650 1653 raft_consensus.cc:493] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.901304 1653 raft_consensus.cc:515] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.902179 1653 leader_election.cc:290] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:17.908820 1653 raft_consensus.cc:493] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.909116 1653 raft_consensus.cc:515] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.909919 1653 leader_election.cc:290] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:17.911062 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 012b81223d0f4ae9b742014e329f11a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012b81223d0f4ae9b742014e329f11a9 already in progress: creating tablet
I20251214 20:52:17.912357 1783 tablet_replica.cc:333] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:17.916602 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9523d72fd91445539d67e1eac2a80afd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.916661 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2ae9dc8ead8403cb7215af8778872f0" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.923352 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52daf1204bf142e7b82852bdec6582fb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.932119 1799 raft_consensus.cc:493] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:17.932442 1799 raft_consensus.cc:515] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.934082 1799 leader_election.cc:290] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:17.937805 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "472e4a5990e64d02b1d52dc6e86d7190" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:17.938153 775 raft_consensus.cc:2468] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.938802 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "472e4a5990e64d02b1d52dc6e86d7190" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.939005 775 raft_consensus.cc:3060] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.942237 775 raft_consensus.cc:2468] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.942950 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52daf1204bf142e7b82852bdec6582fb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:17.912915 1762 leader_election.cc:290] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:17.919953 848 leader_election.cc:343] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:17.958400 848 leader_election.cc:343] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:17.958902 848 leader_election.cc:343] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.959115 848 leader_election.cc:304] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:17.959631 848 leader_election.cc:343] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:17.960005 848 leader_election.cc:343] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:17.933887 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd51f330eb649b998a22a343d993b30" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:17.960655 1760 raft_consensus.cc:2749] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:17.962558 911 raft_consensus.cc:2468] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:17.921634 1202 tablet_bootstrap.cc:492] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:17.962924 1202 ts_tablet_manager.cc:1403] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.297s user 0.003s sys 0.011s
I20251214 20:52:17.963547 991 leader_election.cc:304] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.961068 1783 raft_consensus.cc:2243] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:17.964064 1595 raft_consensus.cc:2804] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:17.964412 1595 raft_consensus.cc:493] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:17.964691 1595 raft_consensus.cc:3060] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.967737 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf37c50e511f454ca4887c7f6ca1a746" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:17.970461 1595 raft_consensus.cc:515] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.973889 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd51f330eb649b998a22a343d993b30" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:17.974162 911 raft_consensus.cc:3060] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:17.977063 911 raft_consensus.cc:2468] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:17.981418 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]). Expected version 1 got 0
I20251214 20:52:17.982041 991 leader_election.cc:304] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:17.982846 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 4a625fcb95324466a12ab118bdb859f1 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:17.983198 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 1 got 0
I20251214 20:52:17.984004 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:17.984541 453 catalog_manager.cc:5654] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:17.985370 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet a142157acbcd44adb6099b4428103ee5 (table default.Table_0870823e1f29409f9d61ebee0bbd060d [id=0a848dd5f6474712b2eba70add3743dc]). Expected version 3 got 0
I20251214 20:52:17.961789 1789 tablet.cc:1722] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:17.985771 453 catalog_manager.cc:5654] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:17.986302 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:17.986683 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8d74d3abc45a41969289c0a0892bfa04 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]). Expected version 1 got 0
I20251214 20:52:17.987001 453 catalog_manager.cc:5654] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:17.987630 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]). Expected version 1 got 0
I20251214 20:52:17.988039 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ca263a80702f4f64aa7c872c2be08207 (table default.Table_642da83f7ed0444688afdf91b7a9e7e8 [id=bca3a2d37f8b4fc797a5c52a43c14869]). Expected version 1 got 0
I20251214 20:52:17.988413 453 catalog_manager.cc:5654] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "074399864fb2470fa9e2ee87c1fe2212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } }
I20251214 20:52:17.988829 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 144dee1a6d7344f2b79da6e79e67599e (table default.Table_6bac3fed78714367b43db88be8ee1712 [id=8cd62da3d57c4be4b5f10ace49d2e148]). Expected version 3 got 0
I20251214 20:52:17.989094 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1]). Expected version 1 got 0
I20251214 20:52:17.989455 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:17.990167 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 0c973e0111804b8b869068bdcc5869bf (table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f]). Expected version 1 got 0
I20251214 20:52:17.991274 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:17.992098 1595 leader_election.cc:290] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:17.993701 1664 raft_consensus.cc:2804] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:17.993997 1664 raft_consensus.cc:697] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:17.994262 1664 consensus_queue.cc:237] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:17.995687 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 012b81223d0f4ae9b742014e329f11a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:18.002070 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2accf8a160584c82bb695a52e3f2b05c (table default.Table_c3fc0fa4ec254ff889443791279bb8af [id=d31b208e2e284575ad429878129a28f5]). Expected version 1 got 0
I20251214 20:52:18.004294 1790 tablet.cc:1722] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:18.004506 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:18.004838 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:18.011113 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:18.012171 1202 raft_consensus.cc:359] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.012511 1202 raft_consensus.cc:385] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.012593 1202 raft_consensus.cc:740] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.012900 1202 consensus_queue.cc:260] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.013293 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:18.013486 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 4d6e7306254044449f1127fb963ca5f0 (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:18.013626 1202 ts_tablet_manager.cc:1434] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.051s user 0.000s sys 0.002s
I20251214 20:52:18.013659 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:18.013873 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:18.015856 1783 raft_consensus.cc:2272] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:18.018002 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 06381efe75284e649fd6d1a9903cfa9b (table default.Table_7bc0f81d22a643589e8a54ceb2477807 [id=db33edd24660463ea648be7da745e29c]). Expected version 1 got 0
I20251214 20:52:18.017498 1783 ts_tablet_manager.cc:1916] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:18.019160 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet cf04f91e79754569a51d468ab576b0dc (table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d]). Expected version 1 got 0
W20251214 20:52:18.018862 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 56224.7us
I20251214 20:52:18.020854 453 catalog_manager.cc:5654] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "074399864fb2470fa9e2ee87c1fe2212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } }
I20251214 20:52:18.021299 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf37c50e511f454ca4887c7f6ca1a746" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.022645 989 leader_election.cc:304] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:18.022938 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]). Expected version 3 got 0
I20251214 20:52:18.023339 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 98b7cac3930a4a9eb16ac413c8f1e092 (table default.Table_e5f3db562155448e91e2e4314254cbe6 [id=826a2666d167435c8fdfd9ab8b8819e6]). Expected version 1 got 0
W20251214 20:52:18.023900 989 leader_election.cc:343] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.024240 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd8226b431c6456885e928880938969d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.024766 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3fa0317e999b483788e5c5652e6e83de (table default.Table_b01c45a1fe8f4a71858fcb2127177471 [id=ef22180b36a241fea13d0e622c02b50a]). Expected version 1 got 0
I20251214 20:52:18.025089 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:18.025331 989 leader_election.cc:304] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:18.025672 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_55fe32e16f364e58afdd57c80a8966ae [id=6157817e96d642d09f9941ad7f7e82ee]). Expected version 1 got 0
I20251214 20:52:18.026034 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ff58f321124d12b5330dca185d426e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.024583 1800 raft_consensus.cc:2804] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:18.026327 1664 raft_consensus.cc:2749] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:18.026638 1800 raft_consensus.cc:697] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:18.027089 1800 consensus_queue.cc:237] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.027544 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:18.029098 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ff58f321124d12b5330dca185d426e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:18.027736 989 leader_election.cc:343] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.030014 1664 consensus_queue.cc:1048] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:18.016223 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 48403.5us
W20251214 20:52:18.030792 989 leader_election.cc:343] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.031760 712 consensus_peers.cc:597] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c25832da5c1644329e18653000f3330e. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:18.032166 989 leader_election.cc:343] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.027973 1203 tablet_bootstrap.cc:492] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:18.033587 1203 ts_tablet_manager.cc:1403] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.357s user 0.130s sys 0.151s
I20251214 20:52:18.034818 1203 raft_consensus.cc:359] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.035044 1203 raft_consensus.cc:385] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.035166 1203 raft_consensus.cc:740] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.035409 1203 consensus_queue.cc:260] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.035969 1203 ts_tablet_manager.cc:1434] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:18.036356 1203 tablet_bootstrap.cc:492] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:18.038733 1203 tablet_bootstrap.cc:654] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.041194 1203 tablet_bootstrap.cc:492] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:18.041405 1203 ts_tablet_manager.cc:1403] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251214 20:52:18.042232 1203 raft_consensus.cc:359] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.042446 1203 raft_consensus.cc:385] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.042519 1203 raft_consensus.cc:740] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.042763 1203 consensus_queue.cc:260] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.043319 1203 ts_tablet_manager.cc:1434] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251214 20:52:18.043728 1203 tablet_bootstrap.cc:492] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:18.045233 1762 raft_consensus.cc:493] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.045434 1762 raft_consensus.cc:515] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.045696 1203 tablet_bootstrap.cc:654] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.046140 1762 leader_election.cc:290] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:18.046805 848 leader_election.cc:343] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.047084 848 leader_election.cc:343] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.047351 848 leader_election.cc:343] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.048328 742 tablet_service.cc:1552] Processing DeleteTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:18.048414 739 tablet_service.cc:1552] Processing DeleteTablet for tablet bd8226b431c6456885e928880938969d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:18.048597 752 tablet_service.cc:1552] Processing DeleteTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f with delete_type TABLET_DATA_DELETED (replaced by tablet f11834cabdad445b8e53cd237d146282) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:18.048691 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da66ec83e9140f7a605f0c86abff468" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.049324 1476 raft_consensus.cc:493] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.049517 1476 raft_consensus.cc:515] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.049716 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd51f330eb649b998a22a343d993b30" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.050108 1476 leader_election.cc:290] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:18.050854 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afd51f330eb649b998a22a343d993b30" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:18.053175 989 leader_election.cc:343] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.053946 1800 consensus_queue.cc:1048] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:18.054102 989 leader_election.cc:343] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.058161 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a625fcb95324466a12ab118bdb859f1" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:18.058377 1056 raft_consensus.cc:2393] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate a2e8fd1cb731446d88f8e510c7a52999 in this term.
W20251214 20:52:18.058954 851 leader_election.cc:343] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.059158 851 leader_election.cc:304] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:18.059695 1762 raft_consensus.cc:2749] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:18.060487 851 leader_election.cc:343] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: bd8226b431c6456885e928880938969d
I20251214 20:52:18.060597 851 leader_election.cc:304] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:18.060770 851 raft_consensus.cc:2686] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251214 20:52:18.061327 851 leader_election.cc:343] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.061460 851 leader_election.cc:304] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:18.061914 1762 raft_consensus.cc:2749] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:18.062583 1804 raft_consensus.cc:493] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.062778 1804 raft_consensus.cc:515] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.063562 1804 leader_election.cc:290] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:18.064354 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a625fcb95324466a12ab118bdb859f1" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.064790 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4626d02ca548f3bf5a1a9142600bff" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.065057 911 raft_consensus.cc:2468] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:18.066984 991 leader_election.cc:304] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.067557 1804 raft_consensus.cc:2804] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:18.067677 1804 raft_consensus.cc:493] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:18.067755 1804 raft_consensus.cc:3060] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:18.070350 848 leader_election.cc:343] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.070855 1804 raft_consensus.cc:515] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.072539 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4626d02ca548f3bf5a1a9142600bff" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:18.072505 1664 consensus_queue.cc:1048] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:18.072727 911 raft_consensus.cc:3060] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.073625 1717 consensus_queue.cc:1048] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:18.075172 1717 consensus_queue.cc:1048] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:18.076419 1804 leader_election.cc:290] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:18.076435 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4626d02ca548f3bf5a1a9142600bff" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.077006 712 leader_election.cc:304] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:18.078022 911 raft_consensus.cc:2468] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:18.078279 1476 raft_consensus.cc:3060] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.078481 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4626d02ca548f3bf5a1a9142600bff" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:18.078538 991 leader_election.cc:304] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.078832 1804 raft_consensus.cc:2804] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:18.079053 1804 raft_consensus.cc:697] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:18.079396 1804 consensus_queue.cc:237] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.081565 1476 raft_consensus.cc:2749] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:18.081642 989 leader_election.cc:343] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.083506 1804 consensus_queue.cc:1048] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:18.084354 989 leader_election.cc:343] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.086920 774 raft_consensus.cc:3060] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.090842 1760 consensus_queue.cc:1048] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:18.097182 1202 tablet_bootstrap.cc:492] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:18.099619 1202 tablet_bootstrap.cc:654] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:18.100725 712 consensus_peers.cc:597] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d768c45bc02b4d108798efbf3f6aaf7d. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:18.105962 1760 raft_consensus.cc:493] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.106206 1203 tablet_bootstrap.cc:492] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:18.106384 1203 ts_tablet_manager.cc:1403] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.063s user 0.033s sys 0.020s
I20251214 20:52:18.106236 1760 raft_consensus.cc:515] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.107389 1760 leader_election.cc:290] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:18.107635 1203 raft_consensus.cc:359] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.107825 1203 raft_consensus.cc:385] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.107923 1203 raft_consensus.cc:740] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.108225 1203 consensus_queue.cc:260] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.111179 1203 ts_tablet_manager.cc:1434] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251214 20:52:18.111501 1203 tablet_bootstrap.cc:492] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:18.113525 1203 tablet_bootstrap.cc:654] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.120265 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da66ec83e9140f7a605f0c86abff468" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:18.122255 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "604b7c6fe3b24daca70711a1990af556" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:18.128988 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_642da83f7ed0444688afdf91b7a9e7e8" } modify_external_catalogs: true
I20251214 20:52:18.129253 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_642da83f7ed0444688afdf91b7a9e7e8" } modify_external_catalogs: true
W20251214 20:52:18.134943 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0976ffcc649461fac74e20b5e402275 has no consensus state
I20251214 20:52:18.135573 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d7581ad6d39142f4a736c05d0e13811c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.136507 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7581ad6d39142f4a736c05d0e13811c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:18.140475 712 consensus_peers.cc:597] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0d661a1bb034053a4634ff537eafc7d. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:18.140537 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "604b7c6fe3b24daca70711a1990af556" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:18.141301 848 leader_election.cc:343] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.144081 851 leader_election.cc:343] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.144249 851 leader_election.cc:304] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:18.144241 1804 raft_consensus.cc:493] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.144462 1804 raft_consensus.cc:515] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.144501 1653 consensus_queue.cc:1048] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:18.144680 851 leader_election.cc:343] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.144752 1650 raft_consensus.cc:2749] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:18.144805 851 leader_election.cc:304] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:18.144974 1653 consensus_queue.cc:1048] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:18.145282 1804 leader_election.cc:290] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.145325 1653 raft_consensus.cc:2749] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:18.146195 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b92b6d5dde2465684d19aee44a83160" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.146315 1804 raft_consensus.cc:493] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.146499 1804 raft_consensus.cc:515] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.146567 911 raft_consensus.cc:2468] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:18.147145 1804 leader_election.cc:290] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.147277 991 leader_election.cc:304] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.147989 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cafed2861e94cd9b0d10d33864ba45d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.148188 910 raft_consensus.cc:2468] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:18.148712 991 leader_election.cc:304] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.148713 1664 raft_consensus.cc:2804] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:18.149072 1664 raft_consensus.cc:493] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:18.149119 1800 raft_consensus.cc:2804] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:18.149176 1664 raft_consensus.cc:3060] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.149199 1800 raft_consensus.cc:493] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:18.149288 1800 raft_consensus.cc:3060] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.152068 1664 raft_consensus.cc:515] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.152798 1664 leader_election.cc:290] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.153906 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b92b6d5dde2465684d19aee44a83160" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:18.154102 910 raft_consensus.cc:3060] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.156962 910 raft_consensus.cc:2468] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:18.157727 991 leader_election.cc:304] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.158128 1667 raft_consensus.cc:2804] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:18.158282 1667 raft_consensus.cc:697] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:18.158564 1667 consensus_queue.cc:237] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.152072 1800 raft_consensus.cc:515] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.160245 1800 leader_election.cc:290] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.161113 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cafed2861e94cd9b0d10d33864ba45d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:18.161378 910 raft_consensus.cc:3060] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.162034 1653 consensus_queue.cc:1048] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:18.164264 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:34819, user_credentials={real_user=slave}} blocked reactor thread for 62278.8us
I20251214 20:52:18.167660 910 raft_consensus.cc:2468] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:18.168236 991 leader_election.cc:304] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.168586 1664 raft_consensus.cc:2804] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:18.168716 1664 raft_consensus.cc:697] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:18.168998 1664 consensus_queue.cc:237] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.177433 1813 tablet.cc:1722] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:18.184494 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b92b6d5dde2465684d19aee44a83160" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.185689 1476 consensus_queue.cc:1048] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251214 20:52:18.188316 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b92b6d5dde2465684d19aee44a83160" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:18.190256 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 012b81223d0f4ae9b742014e329f11a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012b81223d0f4ae9b742014e329f11a9 already in progress: creating tablet
W20251214 20:52:18.190773 989 leader_election.cc:343] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.190702 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet bd8226b431c6456885e928880938969d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd8226b431c6456885e928880938969d already in progress: creating tablet
I20251214 20:52:18.191556 1812 tablet_replica.cc:333] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:18.191802 1812 raft_consensus.cc:2243] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:18.192364 989 leader_election.cc:343] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.195669 1812 raft_consensus.cc:2272] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:18.195977 441 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a0976ffcc649461fac74e20b5e402275 has no consensus state
I20251214 20:52:18.196368 1812 ts_tablet_manager.cc:1916] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:18.199213 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cafed2861e94cd9b0d10d33864ba45d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:18.199586 1804 consensus_queue.cc:1048] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:18.199744 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cafed2861e94cd9b0d10d33864ba45d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:18.200778 989 leader_election.cc:343] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.201184 989 leader_election.cc:343] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.209136 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_2def2ef868b34777b0585c4b95189baa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.209836 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2def2ef868b34777b0585c4b95189baa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.212541 1476 consensus_queue.cc:1048] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251214 20:52:18.216135 1804 raft_consensus.cc:493] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.216320 1804 raft_consensus.cc:515] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.217432 1804 leader_election.cc:290] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.218156 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a10663bd306d40e7a6375d06d69be22d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.218432 911 raft_consensus.cc:2468] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:18.218854 991 leader_election.cc:304] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.219427 1804 raft_consensus.cc:2804] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:18.219590 1804 raft_consensus.cc:493] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:18.219748 1804 raft_consensus.cc:3060] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.222673 1804 raft_consensus.cc:515] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.224104 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a10663bd306d40e7a6375d06d69be22d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:18.224337 915 raft_consensus.cc:3060] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.227442 915 raft_consensus.cc:2468] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:18.228051 991 leader_election.cc:304] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.228483 1804 leader_election.cc:290] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.228708 1804 raft_consensus.cc:2804] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:18.229018 1804 raft_consensus.cc:697] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:18.229154 1203 tablet_bootstrap.cc:492] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:18.229324 1203 ts_tablet_manager.cc:1403] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.118s user 0.009s sys 0.033s
I20251214 20:52:18.229524 1804 consensus_queue.cc:237] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.230201 1203 raft_consensus.cc:359] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.230381 1203 raft_consensus.cc:385] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.230465 1203 raft_consensus.cc:740] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.230700 1203 consensus_queue.cc:260] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.231356 1203 ts_tablet_manager.cc:1434] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:18.231719 1203 tablet_bootstrap.cc:492] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:18.243474 1804 raft_consensus.cc:493] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.243763 1804 raft_consensus.cc:515] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.244733 1804 leader_election.cc:290] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:18.244930 1203 tablet_bootstrap.cc:654] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.246373 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60ae3db968f6412aa0c3952fafe65f88" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.247963 910 raft_consensus.cc:2468] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:18.248576 1783 ts_tablet_manager.cc:1929] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:18.250970 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a10663bd306d40e7a6375d06d69be22d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:18.251726 1206 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.047s sys 0.076s
W20251214 20:52:18.258131 1206 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.542s user 0.047s sys 0.076s
I20251214 20:52:18.259099 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60ae3db968f6412aa0c3952fafe65f88" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.249084 1650 raft_consensus.cc:493] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.260119 1650 raft_consensus.cc:515] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.261096 1812 ts_tablet_manager.cc:1929] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:18.263404 1650 leader_election.cc:290] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:18.263743 1783 log.cc:1199] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/bd8226b431c6456885e928880938969d
I20251214 20:52:18.261003 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a10663bd306d40e7a6375d06d69be22d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:18.266161 1812 log.cc:1199] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/cd88d1e95ee04ef596863e1ae3687de0
I20251214 20:52:18.266535 1809 consensus_queue.cc:1048] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251214 20:52:18.268105 1206 tablet_bootstrap.cc:492] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:18.268275 1783 ts_tablet_manager.cc:1950] T bd8226b431c6456885e928880938969d P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:18.268306 1206 ts_tablet_manager.cc:1403] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.561s user 0.050s sys 0.076s
I20251214 20:52:18.269191 1206 raft_consensus.cc:359] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.269408 1206 raft_consensus.cc:385] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.269480 1206 raft_consensus.cc:740] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.269733 1206 consensus_queue.cc:260] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.270325 1206 ts_tablet_manager.cc:1434] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:18.270678 1206 tablet_bootstrap.cc:492] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:18.272153 1812 ts_tablet_manager.cc:1950] T cd88d1e95ee04ef596863e1ae3687de0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:18.271960 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c0f49e5d78240a296e47eccb3218e4f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:18.274768 848 leader_election.cc:343] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.275537 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet cd88d1e95ee04ef596863e1ae3687de0 (table default.Table_bda8a86e13ba427e94e8d85a318281ad [id=2018af6df0b448bd8996691cb5a1f62c]) successfully deleted
W20251214 20:52:18.275825 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b2e83bca8ed642b6a1b705ef70fde27f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2e83bca8ed642b6a1b705ef70fde27f already in progress: creating tablet
I20251214 20:52:18.277786 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet bd8226b431c6456885e928880938969d (table default.Table_1917701c01bf48c7b2bfc335d05f6db0 [id=6fba2831971a4bffac786611b54a6468]) successfully deleted
I20251214 20:52:18.277895 1206 tablet_bootstrap.cc:654] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.279147 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_26a82e34bb024bd89d6819c80056ba3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.279807 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26a82e34bb024bd89d6819c80056ba3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:18.281145 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet b2e83bca8ed642b6a1b705ef70fde27f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2e83bca8ed642b6a1b705ef70fde27f already in progress: creating tablet
W20251214 20:52:18.282379 989 leader_election.cc:343] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.282409 991 leader_election.cc:304] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.283948 1206 tablet_bootstrap.cc:492] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:18.284232 1206 ts_tablet_manager.cc:1403] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
W20251214 20:52:18.289855 989 leader_election.cc:343] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.290314 989 leader_election.cc:343] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.298822 1202 tablet_bootstrap.cc:492] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:18.299274 1202 ts_tablet_manager.cc:1403] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.202s user 0.002s sys 0.002s
I20251214 20:52:18.300244 1202 raft_consensus.cc:359] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.300532 1202 raft_consensus.cc:385] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.300829 1202 raft_consensus.cc:740] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.301163 1202 consensus_queue.cc:260] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.302407 1202 ts_tablet_manager.cc:1434] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:18.303354 1202 tablet_bootstrap.cc:492] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:18.305353 1202 tablet_bootstrap.cc:654] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.306440 1804 raft_consensus.cc:2804] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:18.309549 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_616b5b16d08b43eb8a04a21a352031fe" } new_table_name: "default.Table_b7bd23d0e40346ca98a98ca28dced554" modify_external_catalogs: true
I20251214 20:52:18.299178 1206 raft_consensus.cc:359] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.311600 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c0f49e5d78240a296e47eccb3218e4f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:18.317387 1664 raft_consensus.cc:493] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:18.314703 851 leader_election.cc:343] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.319000 851 leader_election.cc:304] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:18.319474 1735 raft_consensus.cc:2749] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:18.321646 1206 raft_consensus.cc:385] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.326102 446 master_service.cc:954] ReplaceTablet: received request to replace tablet 9b1a937150ce4ecfa210c8465d682496 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:18.328262 1202 tablet_bootstrap.cc:492] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:18.328464 1804 raft_consensus.cc:493] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:18.328652 1804 raft_consensus.cc:3060] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.328663 1664 raft_consensus.cc:515] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.329388 1664 leader_election.cc:290] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.330197 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da66ec83e9140f7a605f0c86abff468" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.330416 910 raft_consensus.cc:2468] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:18.331522 1804 raft_consensus.cc:515] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.331665 991 leader_election.cc:304] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.332094 1804 leader_election.cc:290] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:18.332293 1804 raft_consensus.cc:2804] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:18.332376 1804 raft_consensus.cc:493] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:18.332472 1804 raft_consensus.cc:3060] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.332928 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60ae3db968f6412aa0c3952fafe65f88" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:18.333109 910 raft_consensus.cc:3060] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.334956 1804 raft_consensus.cc:515] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.335099 1206 raft_consensus.cc:740] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.335480 1206 consensus_queue.cc:260] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.336071 910 raft_consensus.cc:2468] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:18.336536 991 leader_election.cc:304] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.336947 1664 raft_consensus.cc:2804] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:18.337078 1664 raft_consensus.cc:697] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:18.337368 1664 consensus_queue.cc:237] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.338054 1206 ts_tablet_manager.cc:1434] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
W20251214 20:52:18.338680 1837 log.cc:927] Time spent T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20251214 20:52:18.339713 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60ae3db968f6412aa0c3952fafe65f88" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:18.340276 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da66ec83e9140f7a605f0c86abff468" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:18.340862 989 leader_election.cc:343] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.341780 989 leader_election.cc:343] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.342806 1203 tablet_bootstrap.cc:492] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:18.342965 1203 ts_tablet_manager.cc:1403] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.111s user 0.012s sys 0.005s
I20251214 20:52:18.343735 1206 tablet_bootstrap.cc:492] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:18.344079 1202 ts_tablet_manager.cc:1403] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20251214 20:52:18.344594 1735 raft_consensus.cc:493] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.344925 1735 raft_consensus.cc:515] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.343895 1203 raft_consensus.cc:359] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.345381 1203 raft_consensus.cc:385] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.345481 1203 raft_consensus.cc:740] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.345602 1735 leader_election.cc:290] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:18.345734 1203 consensus_queue.cc:260] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.346607 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b22043d81b493fb12116422b1d3fb1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:18.347086 851 leader_election.cc:343] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.347585 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b22043d81b493fb12116422b1d3fb1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.347997 1203 ts_tablet_manager.cc:1434] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20251214 20:52:18.348094 848 leader_election.cc:343] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.348207 848 leader_election.cc:304] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:18.348501 1203 tablet_bootstrap.cc:492] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:18.349386 1735 raft_consensus.cc:2749] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:18.351929 1206 tablet_bootstrap.cc:654] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.354560 1202 raft_consensus.cc:359] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.354804 1202 raft_consensus.cc:385] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.354883 1202 raft_consensus.cc:740] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.355116 1203 tablet_bootstrap.cc:654] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.355965 1202 consensus_queue.cc:260] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.357610 1804 leader_election.cc:290] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.358049 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da66ec83e9140f7a605f0c86abff468" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:18.358225 910 raft_consensus.cc:3060] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.361012 910 raft_consensus.cc:2468] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:18.357127 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2da66ec83e9140f7a605f0c86abff468" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:18.361507 991 leader_election.cc:304] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.361836 1804 raft_consensus.cc:2804] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
W20251214 20:52:18.361886 989 leader_election.cc:343] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.362190 1804 raft_consensus.cc:697] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:18.362545 1804 consensus_queue.cc:237] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.367540 1202 ts_tablet_manager.cc:1434] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20251214 20:52:18.369113 1203 tablet_bootstrap.cc:492] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:18.369341 1203 ts_tablet_manager.cc:1403] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20251214 20:52:18.370261 1203 raft_consensus.cc:359] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.370685 1203 raft_consensus.cc:385] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.371562 1203 raft_consensus.cc:740] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.371927 1203 consensus_queue.cc:260] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.373394 1203 ts_tablet_manager.cc:1434] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:18.375245 1202 tablet_bootstrap.cc:492] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:18.376942 1202 tablet_bootstrap.cc:654] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.377662 1203 tablet_bootstrap.cc:492] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:18.379356 1203 tablet_bootstrap.cc:654] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.389146 754 tablet_service.cc:1505] Processing CreateTablet for tablet 274185a4115b458e8e9a376bdbcc870b (DEFAULT_TABLE table=default.Table_e82b745b92174d32a0d7bc8f779a6558 [id=43a932a17bd545f282a790e6c094a4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.389745 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274185a4115b458e8e9a376bdbcc870b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.394925 752 tablet_service.cc:1505] Processing CreateTablet for tablet 6a000951f61948ac85fdd431f3d162c2 (DEFAULT_TABLE table=default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.395473 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a000951f61948ac85fdd431f3d162c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.395714 884 tablet_service.cc:1505] Processing CreateTablet for tablet 274185a4115b458e8e9a376bdbcc870b (DEFAULT_TABLE table=default.Table_e82b745b92174d32a0d7bc8f779a6558 [id=43a932a17bd545f282a790e6c094a4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.396143 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274185a4115b458e8e9a376bdbcc870b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.396322 446 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 9b1a937150ce4ecfa210c8465d682496
I20251214 20:52:18.396798 446 catalog_manager.cc:6724] ReplaceTablet: tablet 9b1a937150ce4ecfa210c8465d682496 deleted and replaced by tablet f1eb4c3aa654480caaa69dbd5047365c
I20251214 20:52:18.397023 446 master_service.cc:954] ReplaceTablet: received request to replace tablet 4d6e7306254044449f1127fb963ca5f0 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:18.399907 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b1a937150ce4ecfa210c8465d682496 with delete_type TABLET_DATA_DELETED (replaced by tablet f1eb4c3aa654480caaa69dbd5047365c) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:18.399996 892 tablet_service.cc:1505] Processing CreateTablet for tablet 6a000951f61948ac85fdd431f3d162c2 (DEFAULT_TABLE table=default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.400429 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a000951f61948ac85fdd431f3d162c2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:18.400755 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 9b1a937150ce4ecfa210c8465d682496 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b1a937150ce4ecfa210c8465d682496 already in progress: creating tablet
I20251214 20:52:18.401118 742 tablet_service.cc:1505] Processing CreateTablet for tablet feb19ea389af4ea6bb158a0e8aaeb094 (DEFAULT_TABLE table=default.Table_55706259657e4a09aa794297547e9d98 [id=59d9716c5cee42c391018111323c5181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.401507 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feb19ea389af4ea6bb158a0e8aaeb094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.403304 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b1a937150ce4ecfa210c8465d682496 with delete_type TABLET_DATA_DELETED (replaced by tablet f1eb4c3aa654480caaa69dbd5047365c) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:18.404564 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 9b1a937150ce4ecfa210c8465d682496 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b1a937150ce4ecfa210c8465d682496 already in progress: creating tablet
I20251214 20:52:18.405118 876 tablet_service.cc:1505] Processing CreateTablet for tablet feb19ea389af4ea6bb158a0e8aaeb094 (DEFAULT_TABLE table=default.Table_55706259657e4a09aa794297547e9d98 [id=59d9716c5cee42c391018111323c5181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.405213 891 tablet_service.cc:1505] Processing CreateTablet for tablet b69bb4d5979f492683fc13238a668b9f (DEFAULT_TABLE table=default.Table_0dcbd217695546df9d1ca32583980907 [id=487c7f5559f64d13ae2de3da7c3e49a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.405582 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feb19ea389af4ea6bb158a0e8aaeb094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.405897 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b69bb4d5979f492683fc13238a668b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.408213 739 tablet_service.cc:1505] Processing CreateTablet for tablet b69bb4d5979f492683fc13238a668b9f (DEFAULT_TABLE table=default.Table_0dcbd217695546df9d1ca32583980907 [id=487c7f5559f64d13ae2de3da7c3e49a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.408536 756 tablet_service.cc:1505] Processing CreateTablet for tablet 65e79db74fa046698507c83416ee1669 (DEFAULT_TABLE table=default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.408695 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b69bb4d5979f492683fc13238a668b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.408906 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e79db74fa046698507c83416ee1669. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.415731 883 tablet_service.cc:1505] Processing CreateTablet for tablet d424e3387698434f92e84eb1d43f5949 (DEFAULT_TABLE table=default.Table_ee7f67ff1a1c4014a1cc4f9cf7f6613b [id=1597e24e3fff4007ab66b44505c960d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.416159 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d424e3387698434f92e84eb1d43f5949. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.419736 756 tablet_service.cc:1505] Processing CreateTablet for tablet d424e3387698434f92e84eb1d43f5949 (DEFAULT_TABLE table=default.Table_ee7f67ff1a1c4014a1cc4f9cf7f6613b [id=1597e24e3fff4007ab66b44505c960d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.420321 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d424e3387698434f92e84eb1d43f5949. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.427875 891 tablet_service.cc:1505] Processing CreateTablet for tablet 65e79db74fa046698507c83416ee1669 (DEFAULT_TABLE table=default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.428434 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e79db74fa046698507c83416ee1669. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.436503 876 tablet_service.cc:1505] Processing CreateTablet for tablet fc7353593bda4fceaa389c393ddc19c6 (DEFAULT_TABLE table=default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.437371 739 tablet_service.cc:1505] Processing CreateTablet for tablet e6ebfc373fbf41b1920c271a15302484 (DEFAULT_TABLE table=default.Table_26795454cb054bd6bdb4100765bd3c17 [id=1a3349421ffd4a7d933c8301fffb9154]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.437958 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ebfc373fbf41b1920c271a15302484. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.439608 754 tablet_service.cc:1505] Processing CreateTablet for tablet 3b47ca1a03944cf1847d3578856dafa7 (DEFAULT_TABLE table=default.Table_6a61fe3bf1734a888048db0d20abac60 [id=2fd12be2afd34815aff50d4ba78ea3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.442236 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b47ca1a03944cf1847d3578856dafa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.442670 883 tablet_service.cc:1505] Processing CreateTablet for tablet 6c1893b3761c49ceb7cd71b3bd9f2cb4 (DEFAULT_TABLE table=default.Table_250b3222a83c44b08856444499fa612d [id=1ca50be529bd48398643cb6882958f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.438931 892 tablet_service.cc:1505] Processing CreateTablet for tablet e6ebfc373fbf41b1920c271a15302484 (DEFAULT_TABLE table=default.Table_26795454cb054bd6bdb4100765bd3c17 [id=1a3349421ffd4a7d933c8301fffb9154]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.440485 884 tablet_service.cc:1505] Processing CreateTablet for tablet 3b47ca1a03944cf1847d3578856dafa7 (DEFAULT_TABLE table=default.Table_6a61fe3bf1734a888048db0d20abac60 [id=2fd12be2afd34815aff50d4ba78ea3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.441690 742 tablet_service.cc:1505] Processing CreateTablet for tablet 6c1893b3761c49ceb7cd71b3bd9f2cb4 (DEFAULT_TABLE table=default.Table_250b3222a83c44b08856444499fa612d [id=1ca50be529bd48398643cb6882958f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.443822 752 tablet_service.cc:1505] Processing CreateTablet for tablet 4f1d0e555eee4c56be271d14c34335bd (DEFAULT_TABLE table=default.Table_180ac2f089ba44298206726a47570c5b [id=81609d89fec84b2f9b79ca8986447b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.444756 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d0e555eee4c56be271d14c34335bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.447381 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:18.447467 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c1893b3761c49ceb7cd71b3bd9f2cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.448187 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]). Expected version 1 got 0
I20251214 20:52:18.448587 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 4a625fcb95324466a12ab118bdb859f1 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:18.448916 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8d74d3abc45a41969289c0a0892bfa04 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]). Expected version 1 got 0
I20251214 20:52:18.449175 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 1 got 0
I20251214 20:52:18.449566 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet a142157acbcd44adb6099b4428103ee5 (table default.Table_0870823e1f29409f9d61ebee0bbd060d [id=0a848dd5f6474712b2eba70add3743dc]). Expected version 3 got 0
I20251214 20:52:18.449787 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:18.449982 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]). Expected version 1 got 0
I20251214 20:52:18.450264 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:18.450703 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 144dee1a6d7344f2b79da6e79e67599e (table default.Table_6bac3fed78714367b43db88be8ee1712 [id=8cd62da3d57c4be4b5f10ace49d2e148]). Expected version 3 got 0
I20251214 20:52:18.389111 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 274185a4115b458e8e9a376bdbcc870b (DEFAULT_TABLE table=default.Table_e82b745b92174d32a0d7bc8f779a6558 [id=43a932a17bd545f282a790e6c094a4e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.451928 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274185a4115b458e8e9a376bdbcc870b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.395295 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 6a000951f61948ac85fdd431f3d162c2 (DEFAULT_TABLE table=default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.398495 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b1a937150ce4ecfa210c8465d682496 with delete_type TABLET_DATA_DELETED (replaced by tablet f1eb4c3aa654480caaa69dbd5047365c) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.455276 1206 tablet_bootstrap.cc:492] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:18.401634 1029 tablet_service.cc:1505] Processing CreateTablet for tablet feb19ea389af4ea6bb158a0e8aaeb094 (DEFAULT_TABLE table=default.Table_55706259657e4a09aa794297547e9d98 [id=59d9716c5cee42c391018111323c5181]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.455438 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a000951f61948ac85fdd431f3d162c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.455664 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feb19ea389af4ea6bb158a0e8aaeb094. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.456629 1206 ts_tablet_manager.cc:1403] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.113s user 0.016s sys 0.004s
I20251214 20:52:18.457660 1206 raft_consensus.cc:359] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.457979 1206 raft_consensus.cc:385] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.458109 1206 raft_consensus.cc:740] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.458774 1206 consensus_queue.cc:260] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.403491 1030 tablet_service.cc:1505] Processing CreateTablet for tablet b69bb4d5979f492683fc13238a668b9f (DEFAULT_TABLE table=default.Table_0dcbd217695546df9d1ca32583980907 [id=487c7f5559f64d13ae2de3da7c3e49a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.459424 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b69bb4d5979f492683fc13238a668b9f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:18.459931 989 consensus_peers.cc:597] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd88d1e95ee04ef596863e1ae3687de0. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:18.405107 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 65e79db74fa046698507c83416ee1669 (DEFAULT_TABLE table=default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.415050 1031 tablet_service.cc:1505] Processing CreateTablet for tablet d424e3387698434f92e84eb1d43f5949 (DEFAULT_TABLE table=default.Table_ee7f67ff1a1c4014a1cc4f9cf7f6613b [id=1597e24e3fff4007ab66b44505c960d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.435673 1032 tablet_service.cc:1505] Processing CreateTablet for tablet fc7353593bda4fceaa389c393ddc19c6 (DEFAULT_TABLE table=default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.436028 1664 raft_consensus.cc:493] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.437458 1034 tablet_service.cc:1505] Processing CreateTablet for tablet e6ebfc373fbf41b1920c271a15302484 (DEFAULT_TABLE table=default.Table_26795454cb054bd6bdb4100765bd3c17 [id=1a3349421ffd4a7d933c8301fffb9154]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.439738 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 3b47ca1a03944cf1847d3578856dafa7 (DEFAULT_TABLE table=default.Table_6a61fe3bf1734a888048db0d20abac60 [id=2fd12be2afd34815aff50d4ba78ea3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.463863 1206 ts_tablet_manager.cc:1434] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251214 20:52:18.441963 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 6c1893b3761c49ceb7cd71b3bd9f2cb4 (DEFAULT_TABLE table=default.Table_250b3222a83c44b08856444499fa612d [id=1ca50be529bd48398643cb6882958f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.444100 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 4f1d0e555eee4c56be271d14c34335bd (DEFAULT_TABLE table=default.Table_180ac2f089ba44298206726a47570c5b [id=81609d89fec84b2f9b79ca8986447b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.445510 1024 tablet_service.cc:1505] Processing CreateTablet for tablet eb29fd8c85344ed8aa8d880e2ed240bc (DEFAULT_TABLE table=default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 [id=23efc0ee36244ca0a6ff0f2faeb00b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.468041 1206 tablet_bootstrap.cc:492] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:18.468305 1809 raft_consensus.cc:493] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.468552 1809 raft_consensus.cc:515] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.469249 1809 leader_election.cc:290] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:18.469919 752 tablet_service.cc:1505] Processing CreateTablet for tablet eb29fd8c85344ed8aa8d880e2ed240bc (DEFAULT_TABLE table=default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 [id=23efc0ee36244ca0a6ff0f2faeb00b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.470897 889 tablet_service.cc:1505] Processing CreateTablet for tablet 4f1d0e555eee4c56be271d14c34335bd (DEFAULT_TABLE table=default.Table_180ac2f089ba44298206726a47570c5b [id=81609d89fec84b2f9b79ca8986447b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.471277 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c16895ca08d94b56b5c4bc9352e330ef" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.471872 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e79db74fa046698507c83416ee1669. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.472144 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ebfc373fbf41b1920c271a15302484. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.472323 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c16895ca08d94b56b5c4bc9352e330ef" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:18.472903 1054 raft_consensus.cc:2393] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate a2e8fd1cb731446d88f8e510c7a52999 in this term.
I20251214 20:52:18.475409 893 tablet_service.cc:1505] Processing CreateTablet for tablet eb29fd8c85344ed8aa8d880e2ed240bc (DEFAULT_TABLE table=default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 [id=23efc0ee36244ca0a6ff0f2faeb00b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.476361 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7353593bda4fceaa389c393ddc19c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.470781 1206 tablet_bootstrap.cc:654] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.477383 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb29fd8c85344ed8aa8d880e2ed240bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.473387 713 leader_election.cc:304] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:18.478065 1809 raft_consensus.cc:3060] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.478631 1664 raft_consensus.cc:515] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.479317 878 tablet_service.cc:1505] Processing CreateTablet for tablet bc73d590531c409e9f60670435dc8223 (DEFAULT_TABLE table=default.Table_59405cb9326245918d9cf6fb65ddbf40 [id=619272205b724c56ba3d2542c2c5879a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.479800 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b47ca1a03944cf1847d3578856dafa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.480621 1804 consensus_queue.cc:1048] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20251214 20:52:18.482020 1809 raft_consensus.cc:2749] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:18.482625 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b9ca42a5ce4572aebe2a24f4758b12" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.440586 756 tablet_service.cc:1505] Processing CreateTablet for tablet fc7353593bda4fceaa389c393ddc19c6 (DEFAULT_TABLE table=default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.484035 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d0e555eee4c56be271d14c34335bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.484544 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c1893b3761c49ceb7cd71b3bd9f2cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.485848 1735 raft_consensus.cc:493] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.485858 1202 tablet_bootstrap.cc:492] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:18.486788 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7353593bda4fceaa389c393ddc19c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:18.487308 989 leader_election.cc:343] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.487581 1664 leader_election.cc:290] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.487730 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc7353593bda4fceaa389c393ddc19c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.488281 1735 raft_consensus.cc:515] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.488950 1735 leader_election.cc:290] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:18.489112 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d424e3387698434f92e84eb1d43f5949. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.490420 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6c0f445cd7e44769d79ad5566fd8d97" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:18.490402 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6c0f445cd7e44769d79ad5566fd8d97" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.490968 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb29fd8c85344ed8aa8d880e2ed240bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.491307 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b9ca42a5ce4572aebe2a24f4758b12" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.491499 910 raft_consensus.cc:2468] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:18.491919 991 leader_election.cc:304] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:18.492262 1202 ts_tablet_manager.cc:1403] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.000s
I20251214 20:52:18.492506 1664 raft_consensus.cc:2804] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:18.492605 1664 raft_consensus.cc:493] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:18.492694 1664 raft_consensus.cc:3060] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.493083 742 tablet_service.cc:1505] Processing CreateTablet for tablet bc73d590531c409e9f60670435dc8223 (DEFAULT_TABLE table=default.Table_59405cb9326245918d9cf6fb65ddbf40 [id=619272205b724c56ba3d2542c2c5879a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.495941 1595 consensus_queue.cc:1048] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:18.495975 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc73d590531c409e9f60670435dc8223. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.499200 1026 tablet_service.cc:1505] Processing CreateTablet for tablet bc73d590531c409e9f60670435dc8223 (DEFAULT_TABLE table=default.Table_59405cb9326245918d9cf6fb65ddbf40 [id=619272205b724c56ba3d2542c2c5879a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.499693 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc73d590531c409e9f60670435dc8223. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.500272 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb29fd8c85344ed8aa8d880e2ed240bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.500614 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ebfc373fbf41b1920c271a15302484. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:18.500705 851 leader_election.cc:343] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.500922 848 leader_election.cc:343] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.501142 848 leader_election.cc:304] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:18.501638 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc73d590531c409e9f60670435dc8223. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.501847 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b47ca1a03944cf1847d3578856dafa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.502255 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d0e555eee4c56be271d14c34335bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.502558 1202 raft_consensus.cc:359] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.502758 1202 raft_consensus.cc:385] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.502887 1202 raft_consensus.cc:740] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.503038 1762 raft_consensus.cc:2749] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:18.503149 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c1893b3761c49ceb7cd71b3bd9f2cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.505393 1202 consensus_queue.cc:260] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.505970 1202 ts_tablet_manager.cc:1434] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20251214 20:52:18.506374 1202 tablet_bootstrap.cc:492] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:18.508111 1202 tablet_bootstrap.cc:654] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.451110 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1]). Expected version 1 got 0
I20251214 20:52:18.513319 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:18.513643 739 tablet_service.cc:1505] Processing CreateTablet for tablet f945814952b14131ba4e0c67a403a680 (DEFAULT_TABLE table=default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.514640 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:18.514830 882 tablet_service.cc:1505] Processing CreateTablet for tablet f945814952b14131ba4e0c67a403a680 (DEFAULT_TABLE table=default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.514977 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 0c973e0111804b8b869068bdcc5869bf (table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f]). Expected version 1 got 0
I20251214 20:52:18.515357 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f945814952b14131ba4e0c67a403a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.518520 880 tablet_service.cc:1505] Processing CreateTablet for tablet 140667afe42f42659a8cd38ed2517201 (DEFAULT_TABLE table=default.Table_13e60d7ade27439bbcab22ee125a50ba [id=8789a018b1c24bc3956fd11aea52b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.516626 755 tablet_service.cc:1505] Processing CreateTablet for tablet 140667afe42f42659a8cd38ed2517201 (DEFAULT_TABLE table=default.Table_13e60d7ade27439bbcab22ee125a50ba [id=8789a018b1c24bc3956fd11aea52b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.518950 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 140667afe42f42659a8cd38ed2517201. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.519295 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f945814952b14131ba4e0c67a403a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.519487 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 140667afe42f42659a8cd38ed2517201. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.519920 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:18.520222 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 4d6e7306254044449f1127fb963ca5f0 (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:18.520426 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:18.520677 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:18.520836 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:18.522416 747 tablet_service.cc:1505] Processing CreateTablet for tablet fe24e4f4e74040249068505058c15622 (DEFAULT_TABLE table=default.Table_00bef63ea36c4a7585c2832be1454b68 [id=6a279e57008549c593df838211cc9cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.522784 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe24e4f4e74040249068505058c15622. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.526404 886 tablet_service.cc:1505] Processing CreateTablet for tablet fe24e4f4e74040249068505058c15622 (DEFAULT_TABLE table=default.Table_00bef63ea36c4a7585c2832be1454b68 [id=6a279e57008549c593df838211cc9cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.526852 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe24e4f4e74040249068505058c15622. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.528092 1595 raft_consensus.cc:493] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.528271 1595 raft_consensus.cc:515] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.528931 1595 leader_election.cc:290] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.529088 1595 raft_consensus.cc:493] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.529222 1595 raft_consensus.cc:515] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.529739 1595 leader_election.cc:290] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.530736 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b22043d81b493fb12116422b1d3fb1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.530967 911 raft_consensus.cc:2468] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:18.530939 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe4086fc7a4f4db0b7541f06c9833aab" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.531303 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe4086fc7a4f4db0b7541f06c9833aab" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.531466 915 raft_consensus.cc:2468] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
W20251214 20:52:18.531518 989 leader_election.cc:343] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.532040 991 leader_election.cc:304] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:18.532419 991 leader_election.cc:304] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:18.532999 1799 raft_consensus.cc:2804] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:18.533087 1799 raft_consensus.cc:493] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:18.533612 1735 raft_consensus.cc:493] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.533764 1735 raft_consensus.cc:515] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.534435 1735 leader_election.cc:290] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:18.534852 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "324379e3217442c3addcfe58060b9ad7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:18.535377 848 leader_election.cc:343] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.536028 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 06381efe75284e649fd6d1a9903cfa9b (table default.Table_7bc0f81d22a643589e8a54ceb2477807 [id=db33edd24660463ea648be7da745e29c]). Expected version 1 got 0
I20251214 20:52:18.536481 747 tablet_service.cc:1505] Processing CreateTablet for tablet a0976ffcc649461fac74e20b5e402275 (DEFAULT_TABLE table=default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.536708 1799 raft_consensus.cc:3060] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.537107 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0976ffcc649461fac74e20b5e402275. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.538343 1595 raft_consensus.cc:2804] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:18.538462 1595 raft_consensus.cc:493] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:18.538587 1595 raft_consensus.cc:3060] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.539698 1029 tablet_service.cc:1505] Processing CreateTablet for tablet fe24e4f4e74040249068505058c15622 (DEFAULT_TABLE table=default.Table_00bef63ea36c4a7585c2832be1454b68 [id=6a279e57008549c593df838211cc9cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.540124 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe24e4f4e74040249068505058c15622. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.540982 1026 tablet_service.cc:1505] Processing CreateTablet for tablet f945814952b14131ba4e0c67a403a680 (DEFAULT_TABLE table=default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.541332 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f945814952b14131ba4e0c67a403a680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.545637 774 raft_consensus.cc:3060] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.536350 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2accf8a160584c82bb695a52e3f2b05c (table default.Table_c3fc0fa4ec254ff889443791279bb8af [id=d31b208e2e284575ad429878129a28f5]). Expected version 1 got 0
I20251214 20:52:18.547187 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet cf04f91e79754569a51d468ab576b0dc (table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d]). Expected version 1 got 0
I20251214 20:52:18.547776 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 140667afe42f42659a8cd38ed2517201 (DEFAULT_TABLE table=default.Table_13e60d7ade27439bbcab22ee125a50ba [id=8789a018b1c24bc3956fd11aea52b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.548138 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 140667afe42f42659a8cd38ed2517201. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.549187 885 tablet_service.cc:1505] Processing CreateTablet for tablet a0976ffcc649461fac74e20b5e402275 (DEFAULT_TABLE table=default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.549774 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0976ffcc649461fac74e20b5e402275. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.551970 1799 raft_consensus.cc:515] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.547605 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]). Expected version 3 got 0
I20251214 20:52:18.552497 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 98b7cac3930a4a9eb16ac413c8f1e092 (table default.Table_e5f3db562155448e91e2e4314254cbe6 [id=826a2666d167435c8fdfd9ab8b8819e6]). Expected version 1 got 0
I20251214 20:52:18.534847 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b22043d81b493fb12116422b1d3fb1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.554244 888 tablet_service.cc:1505] Processing CreateTablet for tablet 082ec70b8ee646b898d9201ad073d292 (DEFAULT_TABLE table=default.Table_3d23e6dc5242452693b8aaac124a863f [id=8f720514e13b45689817a9811fb624d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.552726 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 3fa0317e999b483788e5c5652e6e83de (table default.Table_b01c45a1fe8f4a71858fcb2127177471 [id=ef22180b36a241fea13d0e622c02b50a]). Expected version 1 got 0
I20251214 20:52:18.554855 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 082ec70b8ee646b898d9201ad073d292. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.555068 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:18.555351 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_55fe32e16f364e58afdd57c80a8966ae [id=6157817e96d642d09f9941ad7f7e82ee]). Expected version 1 got 0
I20251214 20:52:18.555485 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:18.555681 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:18.559322 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet ca263a80702f4f64aa7c872c2be08207
I20251214 20:52:18.559854 449 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 58) took 1042 ms. Trace:
I20251214 20:52:18.559983 449 rpcz_store.cc:276] 1214 20:52:17.517046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.128785 (+611739us) service_pool.cc:225] Handling call
1214 20:52:18.129344 (+ 559us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:18.129491 (+ 147us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:18.129572 (+ 81us) catalog_manager.cc:2824] Locking tablets
1214 20:52:18.257023 (+127451us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:18.315466 (+ 58443us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:18.315570 (+ 104us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:18.559270 (+243700us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:18.559795 (+ 525us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.310925 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.311036 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.311040 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:18.311043 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:18.311195 (+ 152us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.311210 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.311214 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:18.311365 (+ 151us) tablet.cc:699] Row locks acquired
1214 20:52:18.311367 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:18.311405 (+ 38us) write_op.cc:270] Start()
1214 20:52:18.311437 (+ 32us) write_op.cc:276] Timestamp: P: 1765745538311267 usec, L: 0
1214 20:52:18.311441 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.311626 (+ 185us) log.cc:844] Serialized 756 byte log entry
1214 20:52:18.314886 (+ 3260us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.315022 (+ 136us) write_op.cc:301] APPLY: starting
1214 20:52:18.315037 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.315083 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.315108 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.315203 (+ 95us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.315237 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:18.315244 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:18.315320 (+ 76us) log.cc:844] Serialized 29 byte log entry
1214 20:52:18.315364 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.315380 (+ 16us) write_op.cc:454] Released schema lock
1214 20:52:18.315386 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:18.315460 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.315463 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":430,"num_ops":2,"prepare.queue_time_us":53223,"prepare.run_cpu_time_us":867,"prepare.run_wall_time_us":3243,"raft.queue_time_us":122,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":332,"replication_time_us":3420}]]}
I20251214 20:52:18.561750 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_3f7396ac07d649fa8d342ae05765da80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.562305 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f7396ac07d649fa8d342ae05765da80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.564890 881 tablet_service.cc:1552] Processing DeleteTablet for tablet ca263a80702f4f64aa7c872c2be08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:18.565925 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ca263a80702f4f64aa7c872c2be08207 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca263a80702f4f64aa7c872c2be08207 already in progress: creating tablet
I20251214 20:52:18.573500 742 tablet_service.cc:1505] Processing CreateTablet for tablet f11834cabdad445b8e53cd237d146282 (DEFAULT_TABLE table=default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.574651 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f11834cabdad445b8e53cd237d146282. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.575297 739 tablet_service.cc:1552] Processing DeleteTablet for tablet ca263a80702f4f64aa7c872c2be08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:18.581346 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ca263a80702f4f64aa7c872c2be08207 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca263a80702f4f64aa7c872c2be08207 already in progress: creating tablet
I20251214 20:52:18.582981 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "324379e3217442c3addcfe58060b9ad7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:18.583184 989 leader_election.cc:343] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.583915 851 leader_election.cc:343] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.584017 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet ca263a80702f4f64aa7c872c2be08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.584084 851 leader_election.cc:304] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:18.584607 1735 raft_consensus.cc:2749] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:18.557710 887 tablet_service.cc:1505] Processing CreateTablet for tablet f11834cabdad445b8e53cd237d146282 (DEFAULT_TABLE table=default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.588481 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b22043d81b493fb12116422b1d3fb1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:18.589018 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet c9b7149a436a4cab80b39adccf8567f9
I20251214 20:52:18.589638 451 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 48) took 1106 ms. Trace:
I20251214 20:52:18.589785 451 rpcz_store.cc:276] 1214 20:52:17.483361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:17.486919 (+ 3558us) service_pool.cc:225] Handling call
1214 20:52:17.487553 (+ 634us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:17.487692 (+ 139us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:17.492268 (+ 4576us) catalog_manager.cc:2824] Locking tablets
1214 20:52:17.501182 (+ 8914us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:17.977499 (+476317us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:17.977651 (+ 152us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:18.588934 (+611283us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:18.589600 (+ 666us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:17.932643 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:17.932853 (+ 210us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:17.932859 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:17.932873 (+ 14us) tablet.cc:660] Decoding operations
1214 20:52:17.933073 (+ 200us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:17.933094 (+ 21us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:17.933101 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:17.933234 (+ 133us) tablet.cc:699] Row locks acquired
1214 20:52:17.933236 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:17.933279 (+ 43us) write_op.cc:270] Start()
1214 20:52:17.933318 (+ 39us) write_op.cc:276] Timestamp: P: 1765745537933171 usec, L: 0
1214 20:52:17.933322 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:17.933583 (+ 261us) log.cc:844] Serialized 756 byte log entry
1214 20:52:17.972850 (+ 39267us) op_driver.cc:464] REPLICATION: finished
1214 20:52:17.973080 (+ 230us) write_op.cc:301] APPLY: starting
1214 20:52:17.973106 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:17.973182 (+ 76us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:17.973218 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:17.973323 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:17.973376 (+ 53us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:17.973385 (+ 9us) write_op.cc:312] APPLY: finished
1214 20:52:17.973506 (+ 121us) log.cc:844] Serialized 29 byte log entry
1214 20:52:17.973585 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:17.973648 (+ 63us) write_op.cc:454] Released schema lock
1214 20:52:17.973658 (+ 10us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:18.001783 (+ 28125us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.001800 (+ 17us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":161,"apply.run_cpu_time_us":666,"apply.run_wall_time_us":666,"num_ops":2,"prepare.queue_time_us":380794,"prepare.run_cpu_time_us":1361,"prepare.run_wall_time_us":69062,"raft.queue_time_us":174,"raft.run_cpu_time_us":509,"raft.run_wall_time_us":513,"replication_time_us":39480}]]}
I20251214 20:52:18.590550 451 master_service.cc:954] ReplaceTablet: received request to replace tablet 8d74d3abc45a41969289c0a0892bfa04 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:18.591506 739 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b7149a436a4cab80b39adccf8567f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:18.592319 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2accf8a160584c82bb695a52e3f2b05c consensus state has no leader
W20251214 20:52:18.592621 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c9b7149a436a4cab80b39adccf8567f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b7149a436a4cab80b39adccf8567f9 already in progress: creating tablet
W20251214 20:52:18.593254 448 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98105bceda9b44bf893f82a4234a3c81 consensus state has no leader
I20251214 20:52:18.593269 747 tablet_service.cc:1505] Processing CreateTablet for tablet 082ec70b8ee646b898d9201ad073d292 (DEFAULT_TABLE table=default.Table_3d23e6dc5242452693b8aaac124a863f [id=8f720514e13b45689817a9811fb624d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.596256 1595 raft_consensus.cc:515] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.597184 1031 tablet_service.cc:1505] Processing CreateTablet for tablet a0976ffcc649461fac74e20b5e402275 (DEFAULT_TABLE table=default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.598003 1799 leader_election.cc:290] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.597415 1804 consensus_queue.cc:1048] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
W20251214 20:52:18.605481 989 leader_election.cc:343] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.606504 1020 tablet_service.cc:1505] Processing CreateTablet for tablet f11834cabdad445b8e53cd237d146282 (DEFAULT_TABLE table=default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.611764 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f11834cabdad445b8e53cd237d146282. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.613425 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe4086fc7a4f4db0b7541f06c9833aab" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:18.613636 913 raft_consensus.cc:3060] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.614219 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0976ffcc649461fac74e20b5e402275. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:18.617075 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 9b1a937150ce4ecfa210c8465d682496 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b1a937150ce4ecfa210c8465d682496 already in progress: creating tablet
W20251214 20:52:18.617354 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ca263a80702f4f64aa7c872c2be08207 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca263a80702f4f64aa7c872c2be08207 already in progress: creating tablet
I20251214 20:52:18.618264 1735 consensus_queue.cc:1048] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20251214 20:52:18.619248 881 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b7149a436a4cab80b39adccf8567f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:18.620852 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c9b7149a436a4cab80b39adccf8567f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b7149a436a4cab80b39adccf8567f9 already in progress: creating tablet
I20251214 20:52:18.622114 1595 leader_election.cc:290] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.624667 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b22043d81b493fb12116422b1d3fb1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:18.624910 911 raft_consensus.cc:3060] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.625005 1664 raft_consensus.cc:515] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.627803 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 082ec70b8ee646b898d9201ad073d292 (DEFAULT_TABLE table=default.Table_3d23e6dc5242452693b8aaac124a863f [id=8f720514e13b45689817a9811fb624d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:18.629106 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b7149a436a4cab80b39adccf8567f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.627648 1799 consensus_queue.cc:1048] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20251214 20:52:18.631160 1595 raft_consensus.cc:493] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.631426 1202 tablet_bootstrap.cc:492] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:18.631610 1202 ts_tablet_manager.cc:1403] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.125s user 0.000s sys 0.009s
W20251214 20:52:18.633577 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c9b7149a436a4cab80b39adccf8567f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b7149a436a4cab80b39adccf8567f9 already in progress: creating tablet
I20251214 20:52:18.634213 1804 consensus_queue.cc:1048] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20251214 20:52:18.639657 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 082ec70b8ee646b898d9201ad073d292. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:18.640915 1861 log.cc:927] Time spent T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20251214 20:52:18.645893 1664 leader_election.cc:290] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.647046 1595 raft_consensus.cc:515] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.647991 1595 leader_election.cc:290] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.650485 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f11834cabdad445b8e53cd237d146282. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.651693 454 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 62) took 1118 ms. Trace:
I20251214 20:52:18.651875 454 rpcz_store.cc:276] 1214 20:52:17.532680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.293101 (+760421us) service_pool.cc:225] Handling call
1214 20:52:18.309489 (+ 16388us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b182456 00007ff411c8e146 00007ff411c8de75 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:18.309760 (+ 271us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:18.309973 (+ 213us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:18.310187 (+ 214us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:18.310279 (+ 92us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:18.325931 (+ 15652us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:18.651561 (+325630us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.315679 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.315779 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.315785 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:18.315787 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:18.315868 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.315876 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.315881 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:18.315953 (+ 72us) tablet.cc:699] Row locks acquired
1214 20:52:18.315955 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:18.315988 (+ 33us) write_op.cc:270] Start()
1214 20:52:18.316019 (+ 31us) write_op.cc:276] Timestamp: P: 1765745538315851 usec, L: 0
1214 20:52:18.316023 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.316167 (+ 144us) log.cc:844] Serialized 405 byte log entry
1214 20:52:18.324589 (+ 8422us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.325546 (+ 957us) write_op.cc:301] APPLY: starting
1214 20:52:18.325556 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.325585 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.325615 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.325661 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.325687 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:18.325693 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:18.325744 (+ 51us) log.cc:844] Serialized 23 byte log entry
1214 20:52:18.325779 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.325793 (+ 14us) write_op.cc:454] Released schema lock
1214 20:52:18.325797 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":37141760,"child_traces":[["op",{"apply.queue_time_us":923,"apply.run_cpu_time_us":306,"apply.run_wall_time_us":306,"num_ops":1,"prepare.queue_time_us":4905,"prepare.run_cpu_time_us":584,"prepare.run_wall_time_us":583,"raft.queue_time_us":639,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":8,"replication_time_us":8547,"thread_start_us":39428,"threads_started":1}]]}
I20251214 20:52:18.652889 448 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 51) took 1164 ms. Trace:
I20251214 20:52:18.653023 448 rpcz_store.cc:276] 1214 20:52:17.488677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:17.488747 (+ 70us) service_pool.cc:225] Handling call
1214 20:52:17.494140 (+ 5393us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:17.494603 (+ 463us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:17.495805 (+ 1202us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:17.495981 (+ 176us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:17.783663 (+287682us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:18.652806 (+869143us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:17.634627 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:17.634790 (+ 163us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:17.634800 (+ 10us) write_op.cc:435] Acquired schema lock
1214 20:52:17.634802 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:17.634983 (+ 181us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:17.634998 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:17.635003 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:17.635089 (+ 86us) tablet.cc:699] Row locks acquired
1214 20:52:17.635091 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:17.635170 (+ 79us) write_op.cc:270] Start()
1214 20:52:17.635237 (+ 67us) write_op.cc:276] Timestamp: P: 1765745537635119 usec, L: 0
1214 20:52:17.635242 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:17.658426 (+ 23184us) log.cc:844] Serialized 405 byte log entry
1214 20:52:17.681649 (+ 23223us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 000000000098492a 00007ff413306922 00007ff4146e9875 00007ff4146e8399 00007ff4146f07c0 00007ff4146f05cd 0000000000a8970c 00007ff40748e20f 00007ff40749152b 00007ff4074913a8 0000000000a8970c
1214 20:52:17.782416 (+100767us) op_driver.cc:464] REPLICATION: finished
1214 20:52:17.782621 (+ 205us) write_op.cc:301] APPLY: starting
1214 20:52:17.782646 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:17.782724 (+ 78us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:17.782727 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:17.782845 (+ 118us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:17.782897 (+ 52us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:17.782906 (+ 9us) write_op.cc:312] APPLY: finished
1214 20:52:17.783089 (+ 183us) log.cc:844] Serialized 23 byte log entry
1214 20:52:17.783206 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:17.783238 (+ 32us) write_op.cc:454] Released schema lock
1214 20:52:17.783248 (+ 10us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:17.804057 (+ 20809us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:17.804071 (+ 14us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":124,"apply.run_cpu_time_us":718,"apply.run_wall_time_us":719,"mutex_wait_us":3318,"num_ops":1,"prepare.queue_time_us":137971,"prepare.run_cpu_time_us":1774,"prepare.run_wall_time_us":169418,"raft.queue_time_us":95307,"raft.run_cpu_time_us":534,"raft.run_wall_time_us":536,"replication_time_us":147131,"spinlock_wait_cycles":52541696}]]}
I20251214 20:52:18.654183 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_b5f628ed069a4b19996c9b579d7678bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.654767 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5f628ed069a4b19996c9b579d7678bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.654593 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_f1a942f776f74d61922b38c19e43f5df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.655449 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1a942f776f74d61922b38c19e43f5df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.656466 450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 57) took 1139 ms. Trace:
I20251214 20:52:18.656790 1102 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:18.656607 450 rpcz_store.cc:276] 1214 20:52:17.516930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:17.650837 (+133907us) service_pool.cc:225] Handling call
1214 20:52:17.651290 (+ 453us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:17.651538 (+ 248us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:17.651792 (+ 254us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:17.651890 (+ 98us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:18.240259 (+588369us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:18.656409 (+416150us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.211422 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.211537 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.211541 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:18.211544 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:18.211640 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.211654 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.211659 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:18.211742 (+ 83us) tablet.cc:699] Row locks acquired
1214 20:52:18.211744 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:18.211777 (+ 33us) write_op.cc:270] Start()
1214 20:52:18.211809 (+ 32us) write_op.cc:276] Timestamp: P: 1765745538211660 usec, L: 0
1214 20:52:18.211813 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.211984 (+ 171us) log.cc:844] Serialized 405 byte log entry
1214 20:52:18.239587 (+ 27603us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.239746 (+ 159us) write_op.cc:301] APPLY: starting
1214 20:52:18.239763 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.239807 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.239843 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.239900 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.239936 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:18.239943 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:18.240023 (+ 80us) log.cc:844] Serialized 23 byte log entry
1214 20:52:18.240083 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.240104 (+ 21us) write_op.cc:454] Released schema lock
1214 20:52:18.240111 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:18.242470 (+ 2359us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.242477 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":439,"apply.run_wall_time_us":439,"num_ops":1,"prepare.queue_time_us":558970,"prepare.run_cpu_time_us":722,"prepare.run_wall_time_us":31043,"raft.queue_time_us":23830,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":334,"replication_time_us":27743}]]}
I20251214 20:52:18.657419 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_a91064ac3fb94282a02fbc867fdd581c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.658015 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a91064ac3fb94282a02fbc867fdd581c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.656473 455 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 54) took 1150 ms. Trace:
I20251214 20:52:18.659891 449 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 65) took 1119 ms. Trace:
I20251214 20:52:18.659488 455 rpcz_store.cc:276] 1214 20:52:17.506104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:17.562097 (+ 55993us) service_pool.cc:225] Handling call
1214 20:52:17.635718 (+ 73621us) spinlock_profiling.cc:243] Waited 73.2 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b182456 00007ff411c8e146 00007ff411c8de75 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:17.636015 (+ 297us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:17.636273 (+ 258us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:17.636552 (+ 279us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:17.636641 (+ 89us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:18.124642 (+488001us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:18.656340 (+531698us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.001999 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.002250 (+ 251us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.002257 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:18.002260 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:18.002436 (+ 176us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.002458 (+ 22us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.002465 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:18.002591 (+ 126us) tablet.cc:699] Row locks acquired
1214 20:52:18.002594 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:18.002659 (+ 65us) write_op.cc:270] Start()
1214 20:52:18.002713 (+ 54us) write_op.cc:276] Timestamp: P: 1765745538002530 usec, L: 0
1214 20:52:18.002718 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.003083 (+ 365us) log.cc:844] Serialized 405 byte log entry
1214 20:52:18.123385 (+120302us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.123930 (+ 545us) write_op.cc:301] APPLY: starting
1214 20:52:18.123956 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.124008 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.124065 (+ 57us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.124135 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.124214 (+ 79us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:18.124223 (+ 9us) write_op.cc:312] APPLY: finished
1214 20:52:18.124318 (+ 95us) log.cc:844] Serialized 23 byte log entry
1214 20:52:18.124395 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.124418 (+ 23us) write_op.cc:454] Released schema lock
1214 20:52:18.124425 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:18.124507 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.124511 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":168263552,"child_traces":[["op",{"apply.queue_time_us":461,"apply.run_cpu_time_us":574,"apply.run_wall_time_us":574,"num_ops":1,"prepare.queue_time_us":364802,"prepare.run_cpu_time_us":1380,"prepare.run_wall_time_us":53229,"raft.queue_time_us":673,"raft.run_cpu_time_us":416,"raft.run_wall_time_us":417,"replication_time_us":120620}]]}
I20251214 20:52:18.660255 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_7bc0f81d22a643589e8a54ceb2477807" } modify_external_catalogs: true
I20251214 20:52:18.659976 449 rpcz_store.cc:276] 1214 20:52:17.540421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.561546 (+1021125us) service_pool.cc:225] Handling call
1214 20:52:18.651496 (+ 89950us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:18.652098 (+ 602us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:18.659637 (+ 7539us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:18.659748 (+ 111us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:18.659835 (+ 87us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.652785 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.652910 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.652917 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:18.652919 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:18.653017 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.653031 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.653036 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:18.653117 (+ 81us) tablet.cc:699] Row locks acquired
1214 20:52:18.653119 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:18.653160 (+ 41us) write_op.cc:270] Start()
1214 20:52:18.653194 (+ 34us) write_op.cc:276] Timestamp: P: 1765745538653121 usec, L: 0
1214 20:52:18.653199 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.653407 (+ 208us) log.cc:844] Serialized 433 byte log entry
1214 20:52:18.656947 (+ 3540us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.657077 (+ 130us) write_op.cc:301] APPLY: starting
1214 20:52:18.657099 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.657146 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.657170 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.657274 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.657309 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:18.657315 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:18.657392 (+ 77us) log.cc:844] Serialized 29 byte log entry
1214 20:52:18.657464 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.657486 (+ 22us) write_op.cc:454] Released schema lock
1214 20:52:18.657492 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:18.657577 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.657581 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":83,"apply.run_cpu_time_us":492,"apply.run_wall_time_us":492,"num_ops":2,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":749,"prepare.run_wall_time_us":750,"raft.queue_time_us":426,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":231,"replication_time_us":3720}]]}
I20251214 20:52:18.660421 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_7bc0f81d22a643589e8a54ceb2477807" } modify_external_catalogs: true
I20251214 20:52:18.660677 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_893229542bec44e8b903bb0934bd16f5" } modify_external_catalogs: true
I20251214 20:52:18.660805 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_893229542bec44e8b903bb0934bd16f5" } modify_external_catalogs: true
I20251214 20:52:18.663061 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7c6fe3b24daca70711a1990af556 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.664659 1664 raft_consensus.cc:493] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:18.661669 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 7d4626d02ca548f3bf5a1a9142600bff consensus state has no leader
I20251214 20:52:18.664968 1664 raft_consensus.cc:515] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.665874 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.667402 1664 leader_election.cc:290] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:18.668591 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 082ec70b8ee646b898d9201ad073d292. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:18.672387 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 67) took 1131 ms. Trace:
I20251214 20:52:18.674607 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 69) took 1133 ms. Trace:
I20251214 20:52:18.675930 448 rpcz_store.cc:276] 1214 20:52:17.540553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.653910 (+1113357us) service_pool.cc:225] Handling call
1214 20:52:18.654866 (+ 956us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:18.662379 (+ 7513us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:18.672116 (+ 9737us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:18.672230 (+ 114us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:18.672321 (+ 91us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.663707 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.663844 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.663850 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:18.663852 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:18.663965 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.663979 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.663983 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:18.664061 (+ 78us) tablet.cc:699] Row locks acquired
1214 20:52:18.664063 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:18.664102 (+ 39us) write_op.cc:270] Start()
1214 20:52:18.664134 (+ 32us) write_op.cc:276] Timestamp: P: 1765745538664062 usec, L: 0
1214 20:52:18.664137 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.664475 (+ 338us) log.cc:844] Serialized 433 byte log entry
1214 20:52:18.667810 (+ 3335us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.667932 (+ 122us) write_op.cc:301] APPLY: starting
1214 20:52:18.667944 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.667988 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.668032 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.668136 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.668170 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:18.668176 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:18.668243 (+ 67us) log.cc:844] Serialized 29 byte log entry
1214 20:52:18.668289 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.668305 (+ 16us) write_op.cc:454] Released schema lock
1214 20:52:18.668311 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:18.668380 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.668383 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":440,"apply.run_wall_time_us":440,"num_ops":2,"prepare.queue_time_us":819,"prepare.run_cpu_time_us":914,"prepare.run_wall_time_us":914,"raft.queue_time_us":477,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":257,"replication_time_us":3650}]]}
I20251214 20:52:18.675897 450 rpcz_store.cc:276] 1214 20:52:17.540695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.657219 (+1116524us) service_pool.cc:225] Handling call
1214 20:52:18.658108 (+ 889us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:18.666724 (+ 8616us) spinlock_profiling.cc:243] Waited 8.15 ms on lock 0x606000048d78. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 000000000098492a 00007ff40735ecb5 00007ff40735e9e0 00007ff41af7f2ec 00007ff41af78345 00007ff41af70f1d 00007ff41af7beef 00007ff41b18031a 00007ff411c8a7c6 00007ff411c8a4f5 00007ff40e40656e
1214 20:52:18.666943 (+ 219us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:18.674376 (+ 7433us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:18.674477 (+ 101us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:18.674566 (+ 89us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.667490 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.667589 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.667594 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:18.667596 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:18.667669 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.667679 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.667688 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:18.667752 (+ 64us) tablet.cc:699] Row locks acquired
1214 20:52:18.667754 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:18.667781 (+ 27us) write_op.cc:270] Start()
1214 20:52:18.667810 (+ 29us) write_op.cc:276] Timestamp: P: 1765745538667743 usec, L: 0
1214 20:52:18.667814 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.668017 (+ 203us) log.cc:844] Serialized 433 byte log entry
1214 20:52:18.673384 (+ 5367us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.673544 (+ 160us) write_op.cc:301] APPLY: starting
1214 20:52:18.673561 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.673606 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.673641 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.673752 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.673790 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:18.673797 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:18.673874 (+ 77us) log.cc:844] Serialized 29 byte log entry
1214 20:52:18.673936 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.673956 (+ 20us) write_op.cc:454] Released schema lock
1214 20:52:18.673962 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:18.674027 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.674030 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":18733952,"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":488,"apply.run_wall_time_us":487,"num_ops":2,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":615,"prepare.run_wall_time_us":615,"raft.queue_time_us":377,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":5539,"spinlock_wait_cycles":122752}]]}
I20251214 20:52:18.677516 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b3f5ab40d9334de1a194aab3347387d7" } modify_external_catalogs: true
I20251214 20:52:18.678004 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b3f5ab40d9334de1a194aab3347387d7" } modify_external_catalogs: true
I20251214 20:52:18.677860 448 master_service.cc:954] ReplaceTablet: received request to replace tablet 6b4f582d8165439d86d9228c146dc41a from {username='slave'} at 127.0.0.1:50088
W20251214 20:52:18.679742 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 4a625fcb95324466a12ab118bdb859f1 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 4a625fcb95324466a12ab118bdb859f1 consensus state has no leader
W20251214 20:52:18.682340 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d1389716f5ce44f588c21a34c7a7d4c8 consensus state has no leader
I20251214 20:52:18.682904 452 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]). Expected version 1 got 0
W20251214 20:52:18.683789 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:50727 (request call id 180) took 1023 ms (client timeout 1000 ms). Trace:
W20251214 20:52:18.683905 452 rpcz_store.cc:269] 1214 20:52:17.660046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:17.812536 (+152490us) service_pool.cc:225] Handling call
1214 20:52:17.855174 (+ 42638us) spinlock_profiling.cc:243] Waited 41.8 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b17aa08 00007ff411c87ca6 00007ff411c879d5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:18.683702 (+828528us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":96213760}
I20251214 20:52:18.684657 452 master_service.cc:954] ReplaceTablet: received request to replace tablet b2e83bca8ed642b6a1b705ef70fde27f from {username='slave'} at 127.0.0.1:50088
W20251214 20:52:18.688020 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet bf37c50e511f454ca4887c7f6ca1a746 consensus state has no leader
I20251214 20:52:18.693236 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b9ca42a5ce4572aebe2a24f4758b12" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:18.694169 454 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 68) took 1153 ms. Trace:
I20251214 20:52:18.694341 454 rpcz_store.cc:276] 1214 20:52:17.540632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.653925 (+1113293us) service_pool.cc:225] Handling call
1214 20:52:18.655563 (+ 1638us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:18.675681 (+ 20118us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x606000048d78. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 000000000098492a 00007ff40735ecb5 00007ff40735e9e0 00007ff41af7f2ec 00007ff41af78345 00007ff41af70f1d 00007ff41af7beef 00007ff41b18031a 00007ff411c8a7c6 00007ff411c8a4f5 00007ff40e40656e
1214 20:52:18.676490 (+ 809us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:18.693659 (+ 17169us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.693669 (+ 10us) write_op.cc:454] Released schema lock
1214 20:52:18.693789 (+ 120us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:18.693928 (+ 139us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:18.694064 (+ 136us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.677671 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.677880 (+ 209us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.677889 (+ 9us) write_op.cc:435] Acquired schema lock
1214 20:52:18.677892 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:18.678025 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.678046 (+ 21us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.678053 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:18.678161 (+ 108us) tablet.cc:699] Row locks acquired
1214 20:52:18.678164 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:18.678222 (+ 58us) write_op.cc:270] Start()
1214 20:52:18.678271 (+ 49us) write_op.cc:276] Timestamp: P: 1765745538678174 usec, L: 0
1214 20:52:18.678276 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.678553 (+ 277us) log.cc:844] Serialized 433 byte log entry
1214 20:52:18.684059 (+ 5506us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.684224 (+ 165us) write_op.cc:301] APPLY: starting
1214 20:52:18.684246 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.684313 (+ 67us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.684374 (+ 61us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.684910 (+ 536us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.684953 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:18.684960 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:18.685070 (+ 110us) log.cc:844] Serialized 29 byte log entry
1214 20:52:18.685148 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.685172 (+ 24us) write_op.cc:454] Released schema lock
1214 20:52:18.685179 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":42355968,"child_traces":[["op",{"apply.queue_time_us":99,"apply.run_cpu_time_us":706,"apply.run_wall_time_us":1018,"num_ops":2,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":1050,"prepare.run_wall_time_us":1545,"raft.queue_time_us":547,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":379,"replication_time_us":5745}]]}
I20251214 20:52:18.695627 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.695749 1203 tablet_bootstrap.cc:492] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:18.695376 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 06381efe75284e649fd6d1a9903cfa9b
I20251214 20:52:18.697794 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_989a947ef29f422cbcc7fd84b5cac678"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.698432 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_989a947ef29f422cbcc7fd84b5cac678 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.698374 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_9fa9404adcd247179c0695d6a85e87ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:18.698890 1202 raft_consensus.cc:359] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.699491 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 06381efe75284e649fd6d1a9903cfa9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:18.701373 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fa9404adcd247179c0695d6a85e87ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.700774 910 raft_consensus.cc:3060] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:18.703362 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 06381efe75284e649fd6d1a9903cfa9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06381efe75284e649fd6d1a9903cfa9b already in progress: creating tablet
I20251214 20:52:18.707160 446 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 4d6e7306254044449f1127fb963ca5f0
I20251214 20:52:18.707664 446 catalog_manager.cc:6724] ReplaceTablet: tablet 4d6e7306254044449f1127fb963ca5f0 deleted and replaced by tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f
I20251214 20:52:18.707813 446 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 64) took 1167 ms. Trace:
I20251214 20:52:18.707908 446 rpcz_store.cc:276] 1214 20:52:17.540293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.396962 (+856669us) service_pool.cc:225] Handling call
1214 20:52:18.707752 (+310790us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.691848 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.691952 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.691959 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:18.691961 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:18.692066 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.692077 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.692081 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:18.692186 (+ 105us) tablet.cc:699] Row locks acquired
1214 20:52:18.692189 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:18.692226 (+ 37us) write_op.cc:270] Start()
1214 20:52:18.692261 (+ 35us) write_op.cc:276] Timestamp: P: 1765745538692180 usec, L: 0
1214 20:52:18.692264 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.692442 (+ 178us) log.cc:844] Serialized 622 byte log entry
1214 20:52:18.706170 (+ 13728us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.706350 (+ 180us) write_op.cc:301] APPLY: starting
1214 20:52:18.706370 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.706420 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.706452 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.706572 (+ 120us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.706629 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:18.706636 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:18.706715 (+ 79us) log.cc:844] Serialized 29 byte log entry
1214 20:52:18.706782 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.706802 (+ 20us) write_op.cc:454] Released schema lock
1214 20:52:18.706810 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:18.706899 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.706904 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":124,"apply.run_cpu_time_us":532,"apply.run_wall_time_us":531,"num_ops":2,"prepare.queue_time_us":149,"prepare.run_cpu_time_us":698,"prepare.run_wall_time_us":5778,"raft.queue_time_us":699,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":7,"replication_time_us":13874,"spinlock_wait_cycles":1280}]]}
I20251214 20:52:18.709323 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_3bc5f340978c453190d06f9e7a87f65c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.710076 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bc5f340978c453190d06f9e7a87f65c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.710538 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_493f60b9d8024509856ed934d379f829"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.711222 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_493f60b9d8024509856ed934d379f829 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.707953 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_181aab73cbf1461b85364fc893700db7" } new_table_name: "default.Table_7923a9cba4ce4b27a12617e394ffbf2e" modify_external_catalogs: true
I20251214 20:52:18.715631 1202 raft_consensus.cc:385] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.715801 1202 raft_consensus.cc:740] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.719228 1809 raft_consensus.cc:493] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.719504 1809 raft_consensus.cc:515] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.720454 1809 leader_election.cc:290] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:18.724390 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_8ca3b0a5a22a425b866b0b2565f80fac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.725178 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ca3b0a5a22a425b866b0b2565f80fac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.730942 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 8d74d3abc45a41969289c0a0892bfa04
I20251214 20:52:18.731480 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet b1f4ea72c6274569a98cab6afdc55d55
I20251214 20:52:18.733421 451 catalog_manager.cc:6724] ReplaceTablet: tablet 8d74d3abc45a41969289c0a0892bfa04 deleted and replaced by tablet 727e7b61ae3942098712720d344c5310
I20251214 20:52:18.733567 451 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 66) took 1193 ms. Trace:
I20251214 20:52:18.733655 451 rpcz_store.cc:276] 1214 20:52:17.540488 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.590467 (+1049979us) service_pool.cc:225] Handling call
1214 20:52:18.733504 (+143037us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.709960 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.710097 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.710102 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:18.710105 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:18.710523 (+ 418us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.710532 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.710537 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:18.710606 (+ 69us) tablet.cc:699] Row locks acquired
1214 20:52:18.710608 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:18.710645 (+ 37us) write_op.cc:270] Start()
1214 20:52:18.710671 (+ 26us) write_op.cc:276] Timestamp: P: 1765745538710592 usec, L: 0
1214 20:52:18.710674 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.710821 (+ 147us) log.cc:844] Serialized 622 byte log entry
1214 20:52:18.729636 (+ 18815us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.730360 (+ 724us) write_op.cc:301] APPLY: starting
1214 20:52:18.730369 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.730397 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.730427 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.730508 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.730557 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:18.730563 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:18.730615 (+ 52us) log.cc:844] Serialized 29 byte log entry
1214 20:52:18.730653 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.730667 (+ 14us) write_op.cc:454] Released schema lock
1214 20:52:18.730672 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:18.730733 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.730737 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":693,"apply.run_cpu_time_us":366,"apply.run_wall_time_us":366,"num_ops":2,"prepare.queue_time_us":3041,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":1495,"raft.queue_time_us":484,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":8,"replication_time_us":18947}]]}
I20251214 20:52:18.734546 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b86f4adaf4540f48a4f42bc45e259e5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.735327 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 69f6108792fa4c2a9744e3248c76251f
I20251214 20:52:18.736016 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.736797 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff with delete_type TABLET_DATA_DELETED (replaced by tablet 6a000951f61948ac85fdd431f3d162c2) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.739429 1203 ts_tablet_manager.cc:1403] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.321s user 0.007s sys 0.002s
I20251214 20:52:18.740968 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe4086fc7a4f4db0b7541f06c9833aab" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:18.741425 777 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8af99e0bd8774a678575847a9ec2e557" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.743881 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 06381efe75284e649fd6d1a9903cfa9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:18.743976 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d74d3abc45a41969289c0a0892bfa04 with delete_type TABLET_DATA_DELETED (replaced by tablet 727e7b61ae3942098712720d344c5310) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:18.744316 823 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:18.744371 1812 tablet_replica.cc:333] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:18.744580 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b9ca42a5ce4572aebe2a24f4758b12" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:18.746116 989 leader_election.cc:343] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.746376 989 leader_election.cc:343] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:18.747797 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 06381efe75284e649fd6d1a9903cfa9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06381efe75284e649fd6d1a9903cfa9b already in progress: creating tablet
W20251214 20:52:18.749262 989 leader_election.cc:343] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.749311 1664 consensus_queue.cc:1048] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251214 20:52:18.750414 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af99e0bd8774a678575847a9ec2e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.750612 1856 tablet_replica.cc:333] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:18.750831 1856 raft_consensus.cc:2243] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:18.757607 1856 raft_consensus.cc:2272] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:18.758195 1856 ts_tablet_manager.cc:1916] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:18.760669 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 52daf1204bf142e7b82852bdec6582fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.762683 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a2143b5508445492b970d2588cf1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.763376 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.763619 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a7fc15882e2474bb6885070be13b1fa with delete_type TABLET_DATA_DELETED (replaced by tablet 65e79db74fa046698507c83416ee1669) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:18.765970 448 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e16fe751f10f4616a00d0af811998729 consensus state has no leader
I20251214 20:52:18.766155 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_7041fb3e6d4040f08fd9a554a30e315d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:18.767194 452 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8b86c42f7826440982c88a8625559146" } modify_external_catalogs: true
I20251214 20:52:18.767407 452 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8b86c42f7826440982c88a8625559146" } modify_external_catalogs: true
W20251214 20:52:18.767604 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7041fb3e6d4040f08fd9a554a30e315d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:18.779553 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b86f4adaf4540f48a4f42bc45e259e5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.780511 756 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:18.780786 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d6e7306254044449f1127fb963ca5f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:18.780961 1812 raft_consensus.cc:2243] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:18.781002 1206 tablet_bootstrap.cc:492] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:18.781203 1812 raft_consensus.cc:2272] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:18.781208 1206 ts_tablet_manager.cc:1403] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.313s user 0.053s sys 0.124s
I20251214 20:52:18.782445 1812 ts_tablet_manager.cc:1916] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:18.782438 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 63497e77baeb4d328be2421b5732a28a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.782835 1206 raft_consensus.cc:359] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.783039 1206 raft_consensus.cc:385] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.783633 1203 raft_consensus.cc:359] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.783957 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 69f6108792fa4c2a9744e3248c76251f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:18.784271 1206 raft_consensus.cc:740] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.784550 1206 consensus_queue.cc:260] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:18.785128 1206 ts_tablet_manager.cc:1434] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251214 20:52:18.785518 1206 tablet_bootstrap.cc:492] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:18.786558 1863 log.cc:927] Time spent T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20251214 20:52:18.791710 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d6e7306254044449f1127fb963ca5f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:18.791970 1735 consensus_queue.cc:1048] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20251214 20:52:18.792855 448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_e5f3db562155448e91e2e4314254cbe6" } modify_external_catalogs: true
I20251214 20:52:18.793071 448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_e5f3db562155448e91e2e4314254cbe6" } modify_external_catalogs: true
I20251214 20:52:18.794121 894 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:18.794351 848 consensus_peers.cc:597] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:18.795429 909 raft_consensus.cc:2468] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:18.800302 1202 consensus_queue.cc:260] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:18.801415 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 4d6e7306254044449f1127fb963ca5f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d6e7306254044449f1127fb963ca5f0 already in progress: creating tablet
I20251214 20:52:18.801658 1203 raft_consensus.cc:385] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251214 20:52:18.801697 989 leader_election.cc:343] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.802222 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet f0d661a1bb034053a4634ff537eafc7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.805138 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 2da66ec83e9140f7a605f0c86abff468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.805387 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.805441 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.805707 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet bb698ff30fcc47d09000cc68d2c5225b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.805873 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet c25832da5c1644329e18653000f3330e with delete_type TABLET_DATA_DELETED (replaced by tablet 0c26d1f969ed4071a2f2cf1615a17cac) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.806018 1203 raft_consensus.cc:740] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.806330 1203 consensus_queue.cc:260] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.806921 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 9523d72fd91445539d67e1eac2a80afd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.810707 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet b652a8a80246443491551e76e68d2a8b with delete_type TABLET_DATA_DELETED (replaced by tablet 18e119bc44874aecb4e83c8d6040c8cd) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:18.812915 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 8d74d3abc45a41969289c0a0892bfa04 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 8d74d3abc45a41969289c0a0892bfa04 consensus state has no leader
I20251214 20:52:18.734627 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8af99e0bd8774a678575847a9ec2e557" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:18.815443 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet ca263a80702f4f64aa7c872c2be08207 (table default.Table_642da83f7ed0444688afdf91b7a9e7e8 [id=bca3a2d37f8b4fc797a5c52a43c14869], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ca263a80702f4f64aa7c872c2be08207 consensus state has no leader
W20251214 20:52:18.816771 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet b1f4ea72c6274569a98cab6afdc55d55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1f4ea72c6274569a98cab6afdc55d55 already in progress: creating tablet
I20251214 20:52:18.817472 446 master_service.cc:954] ReplaceTablet: received request to replace tablet 9d89ee602c524c96a21c0a235af07481 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:18.819034 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d74d3abc45a41969289c0a0892bfa04 with delete_type TABLET_DATA_DELETED (replaced by tablet 727e7b61ae3942098712720d344c5310) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:18.819458 1203 ts_tablet_manager.cc:1434] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.066s user 0.003s sys 0.000s
I20251214 20:52:18.822170 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b4f582d8165439d86d9228c146dc41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:18.816834 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 2d164747cb294bd4b126b4420231867d consensus state has no leader
I20251214 20:52:18.823117 1203 tablet_bootstrap.cc:492] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:18.823365 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet cbbfcef827ee4d548939aaf4783c8109 consensus state has no leader
W20251214 20:52:18.824033 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 0c973e0111804b8b869068bdcc5869bf (table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0c973e0111804b8b869068bdcc5869bf consensus state has no leader
W20251214 20:52:18.824331 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3bd5c18696f54ea190c1ef84b93c4f63 consensus state has no leader
I20251214 20:52:18.824510 991 leader_election.cc:304] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
W20251214 20:52:18.826606 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 8fcdd6bafa5c4aa6b59668cde523d010 consensus state has no leader
I20251214 20:52:18.826895 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d6e7306254044449f1127fb963ca5f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:18.828493 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 719d157ac8f74eb7911df05ab5e4a3d5 consensus state has no leader
W20251214 20:52:18.829641 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 37a33cc9bb784f72956ac9f5493a0c3d consensus state has no leader
I20251214 20:52:18.830708 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 06381efe75284e649fd6d1a9903cfa9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.831223 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.831457 1664 raft_consensus.cc:2804] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:18.832382 1664 raft_consensus.cc:493] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:18.832510 1664 raft_consensus.cc:3060] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:18.835686 1203 tablet_bootstrap.cc:654] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.836333 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 69f6108792fa4c2a9744e3248c76251f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:18.840114 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d74d3abc45a41969289c0a0892bfa04 with delete_type TABLET_DATA_DELETED (replaced by tablet 727e7b61ae3942098712720d344c5310) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.842761 908 raft_consensus.cc:2468] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:18.842926 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.843415 1783 tablet_replica.cc:333] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
W20251214 20:52:18.843835 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 5f4760585baf4b4ab3b9b94b5a748a61 consensus state has no leader
W20251214 20:52:18.845089 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 4d6e7306254044449f1127fb963ca5f0 (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 4d6e7306254044449f1127fb963ca5f0 consensus state has no leader
I20251214 20:52:18.845896 1202 ts_tablet_manager.cc:1434] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.187s user 0.000s sys 0.003s
W20251214 20:52:18.846036 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 453cf734f8b34b59aa246d618ffeade5 consensus state has no leader
I20251214 20:52:18.846488 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 69f6108792fa4c2a9744e3248c76251f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:18.847018 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 658e4fe0d188483b9ceb9eb3b3f20835 consensus state has no leader
I20251214 20:52:18.848383 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 324379e3217442c3addcfe58060b9ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:18.851861 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 06381efe75284e649fd6d1a9903cfa9b (table default.Table_7bc0f81d22a643589e8a54ceb2477807 [id=db33edd24660463ea648be7da745e29c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 06381efe75284e649fd6d1a9903cfa9b consensus state has no leader
I20251214 20:52:18.850135 991 leader_election.cc:304] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
W20251214 20:52:18.853420 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet cf04f91e79754569a51d468ab576b0dc (table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet cf04f91e79754569a51d468ab576b0dc consensus state has no leader
I20251214 20:52:18.854987 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b01c45a1fe8f4a71858fcb2127177471" } new_table_name: "default.Table_6c8f53a62dca43e2b8189d4a1fda9989" modify_external_catalogs: true
W20251214 20:52:18.855212 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 98b7cac3930a4a9eb16ac413c8f1e092 (table default.Table_e5f3db562155448e91e2e4314254cbe6 [id=826a2666d167435c8fdfd9ab8b8819e6], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 98b7cac3930a4a9eb16ac413c8f1e092 consensus state has no leader
I20251214 20:52:18.857107 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_bb6020a9d77b42b5955933a449edc238" } new_table_name: "default.Table_66472fe7b4dd4e3eb07839fe78a7bead" modify_external_catalogs: true
I20251214 20:52:18.860067 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ecba341e009147b9b843be246ea2cd7c" } new_table_name: "default.Table_78f54ddbc4d7430fa694c4ce272be743" modify_external_catalogs: true
I20251214 20:52:18.788350 1206 tablet_bootstrap.cc:654] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:18.864005 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet ca57963bcfb54eec9fcbed82757d8387 consensus state has no leader
I20251214 20:52:18.789820 1812 ts_tablet_manager.cc:1929] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:18.864729 1812 log.cc:1199] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/8d74d3abc45a41969289c0a0892bfa04
W20251214 20:52:18.864792 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_55fe32e16f364e58afdd57c80a8966ae [id=6157817e96d642d09f9941ad7f7e82ee], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 07a8662a634a4befbd2d6251000d2059 consensus state has no leader
I20251214 20:52:18.865281 1812 ts_tablet_manager.cc:1950] T 8d74d3abc45a41969289c0a0892bfa04 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
W20251214 20:52:18.865437 453 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b70579ddf96f4499b4e4f235e8f69d67 consensus state has no leader
I20251214 20:52:18.813838 1717 consensus_queue.cc:1048] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:18.866485 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:46855 (request call id 160) took 1222 ms (client timeout 1000 ms). Trace:
W20251214 20:52:18.866608 453 rpcz_store.cc:269] 1214 20:52:17.644452 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:17.649279 (+ 4827us) service_pool.cc:225] Handling call
1214 20:52:18.866444 (+1217165us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.242664 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.242792 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.242797 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:18.242800 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:18.242987 (+ 187us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.243000 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.243006 (+ 6us) tablet.cc:683] Acquiring locks for 5 operations
1214 20:52:18.243169 (+ 163us) tablet.cc:699] Row locks acquired
1214 20:52:18.243171 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:18.245372 (+ 2201us) write_op.cc:270] Start()
1214 20:52:18.245421 (+ 49us) write_op.cc:276] Timestamp: P: 1765745538243079 usec, L: 0
1214 20:52:18.245426 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.247650 (+ 2224us) log.cc:844] Serialized 2024 byte log entry
1214 20:52:18.252957 (+ 5307us) op_driver.cc:464] REPLICATION: finished
1214 20:52:18.253112 (+ 155us) write_op.cc:301] APPLY: starting
1214 20:52:18.253129 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:18.253183 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:18.253219 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:18.253325 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:18.253380 (+ 55us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
1214 20:52:18.253386 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:18.253471 (+ 85us) log.cc:844] Serialized 47 byte log entry
1214 20:52:18.253528 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.253553 (+ 25us) write_op.cc:454] Released schema lock
1214 20:52:18.253560 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:18.256981 (+ 3421us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:18.256987 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7791,"reported_tablets":137,"child_traces":[["op",{"apply.queue_time_us":107,"apply.run_cpu_time_us":521,"apply.run_wall_time_us":521,"num_ops":5,"prepare.queue_time_us":213744,"prepare.run_cpu_time_us":1047,"prepare.run_wall_time_us":14309,"raft.queue_time_us":795,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":271,"replication_time_us":7502}]]}
I20251214 20:52:18.818579 907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4626d02ca548f3bf5a1a9142600bff" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:18.870244 907 raft_consensus.cc:2393] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:18.870816 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_3b13fb90bb2c4a0db268b7777b08f240"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:18.871507 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b13fb90bb2c4a0db268b7777b08f240 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:18.876675 848 consensus_peers.cc:597] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:18.879906 913 raft_consensus.cc:2468] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:18.882378 1783 raft_consensus.cc:2243] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:18.885377 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_2def2ef868b34777b0585c4b95189baa" } new_table_name: "default.Table_e26cdfc4ab8f4f19b52de34448bd70e5" modify_external_catalogs: true
I20251214 20:52:18.886715 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 8d74d3abc45a41969289c0a0892bfa04 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]) successfully deleted
W20251214 20:52:18.887316 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b1f4ea72c6274569a98cab6afdc55d55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1f4ea72c6274569a98cab6afdc55d55 already in progress: creating tablet
I20251214 20:52:18.888085 910 raft_consensus.cc:2468] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
W20251214 20:52:18.889495 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 4d6e7306254044449f1127fb963ca5f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d6e7306254044449f1127fb963ca5f0 already in progress: creating tablet
W20251214 20:52:18.890067 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 69f6108792fa4c2a9744e3248c76251f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f6108792fa4c2a9744e3248c76251f already in progress: creating tablet
I20251214 20:52:18.897334 1783 raft_consensus.cc:2272] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:18.901307 1783 ts_tablet_manager.cc:1916] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:18.902209 1202 tablet_bootstrap.cc:492] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:18.908825 1885 tablet.cc:1722] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251214 20:52:18.912518 911 raft_consensus.cc:2468] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:18.951640 1202 tablet_bootstrap.cc:654] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:18.954824 1783 ts_tablet_manager.cc:1929] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:18.955114 1783 log.cc:1199] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/8d74d3abc45a41969289c0a0892bfa04
I20251214 20:52:18.955952 1783 ts_tablet_manager.cc:1950] T 8d74d3abc45a41969289c0a0892bfa04 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:18.960795 1735 raft_consensus.cc:493] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:18.961004 1735 raft_consensus.cc:515] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.961781 1735 leader_election.cc:290] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:18.963019 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bbb28259c1442ae94f4eff76c25c539" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:18.964016 1889 tablet.cc:1722] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20251214 20:52:18.964771 848 leader_election.cc:343] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:18.973450 1202 tablet_bootstrap.cc:492] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:18.973773 1202 ts_tablet_manager.cc:1403] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.003s
I20251214 20:52:18.975749 1202 raft_consensus.cc:359] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.976370 1202 raft_consensus.cc:385] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.976747 1202 raft_consensus.cc:740] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.977102 1202 consensus_queue.cc:260] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:18.978364 1202 ts_tablet_manager.cc:1434] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:18.978992 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 8d74d3abc45a41969289c0a0892bfa04 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]) successfully deleted
W20251214 20:52:18.979396 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 69f6108792fa4c2a9744e3248c76251f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f6108792fa4c2a9744e3248c76251f already in progress: creating tablet
I20251214 20:52:18.984396 1206 tablet_bootstrap.cc:492] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:18.984700 1206 ts_tablet_manager.cc:1403] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.199s user 0.001s sys 0.007s
I20251214 20:52:18.985842 1206 raft_consensus.cc:359] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.986195 1206 raft_consensus.cc:385] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:18.986322 1206 raft_consensus.cc:740] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:18.986649 1206 consensus_queue.cc:260] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:18.994835 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b22043d81b493fb12116422b1d3fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet ba7952290e2c4746802cffeb14971da0) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.995332 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet a063083f54ad49d2a842fa080d44ed85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.996027 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:18.996294 1202 tablet_bootstrap.cc:492] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:18.998687 1202 tablet_bootstrap.cc:654] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.000928 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd6856b800247de896f16eee66f2827 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c3962c33a294fa98f8bd485268aee71) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.001322 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet bd8226b431c6456885e928880938969d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.023803 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.024277 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet ba50e32f7e724868bb891045a939e362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.024150 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 84a0237e4ee34f94821d8647a400575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.028067 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.029564 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.029651 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.030388 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.032083 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 1dca369dc5f54a4ea1783e334c5670d6) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.032277 1799 raft_consensus.cc:2804] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:19.032727 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet ec85857f047e44d78835305ddcb70768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.032842 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 569a02142c71427d91868b02695ae12b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.032938 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 649f5bd67cbb4f61b7066269f0bf9266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.033531 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet b315744c8cd043c5a8bdceec73a7e28b with delete_type TABLET_DATA_DELETED (replaced by tablet f945814952b14131ba4e0c67a403a680) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.033861 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet ee7d709316b24b8eb223a838710c2979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.033998 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.034373 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.034528 1799 raft_consensus.cc:493] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:19.035051 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f with delete_type TABLET_DATA_DELETED (replaced by tablet f11834cabdad445b8e53cd237d146282) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.035638 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 40a9a6192c744282ab5672cbe15fd299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.036000 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.036826 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 35a529702f924a5c9bf53c3c668c6094 with delete_type TABLET_DATA_DELETED (replaced by tablet 719d157ac8f74eb7911df05ab5e4a3d5) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.037686 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3962c33a294fa98f8bd485268aee71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.038118 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet a10663bd306d40e7a6375d06d69be22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.038127 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf with delete_type TABLET_DATA_DELETED (replaced by tablet 6fef1d6add764fd28a8f4995b337ea71) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.038486 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff58f321124d12b5330dca185d426e with delete_type TABLET_DATA_DELETED (replaced by tablet 42bdcecba5534294aca1eb419f18e9d2) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.039929 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet b1df0dad2fb040a5a12e0401e00c87db with delete_type TABLET_DATA_DELETED (replaced by tablet 8fcdd6bafa5c4aa6b59668cde523d010) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.040040 1799 raft_consensus.cc:3060] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:19.040458 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet c16895ca08d94b56b5c4bc9352e330ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.039976 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 012b81223d0f4ae9b742014e329f11a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.050051 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet a0279992d1f2419ab1caae116a0391b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.054852 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:19.059932 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 604b7c6fe3b24daca70711a1990af556 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 604b7c6fe3b24daca70711a1990af556 already in progress: creating tablet
W20251214 20:52:19.060227 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3bbb28259c1442ae94f4eff76c25c539 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bbb28259c1442ae94f4eff76c25c539 already in progress: creating tablet
I20251214 20:52:19.061192 1891 raft_consensus.cc:493] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.061359 1891 raft_consensus.cc:515] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:19.062325 1891 leader_election.cc:290] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:19.065546 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60ae3db968f6412aa0c3952fafe65f88" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:19.065878 911 raft_consensus.cc:2393] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:19.071259 1664 raft_consensus.cc:515] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.071280 1898 tablet.cc:1722] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251214 20:52:19.073977 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 184177us
I20251214 20:52:19.075340 1650 raft_consensus.cc:493] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.075584 1650 raft_consensus.cc:515] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:19.076325 1650 leader_election.cc:290] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:19.077447 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f4760585baf4b4ab3b9b94b5a748a61" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:19.077771 1664 leader_election.cc:290] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:19.078015 1799 raft_consensus.cc:515] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:19.078073 848 leader_election.cc:343] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.080142 1889 tablet.cc:1722] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:19.100421 1206 ts_tablet_manager.cc:1434] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.116s user 0.001s sys 0.001s
W20251214 20:52:19.108525 454 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d7c7b3f62764ff497f4d70b53eed7de consensus state has no leader
I20251214 20:52:19.115491 454 catalog_manager.cc:5654] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:19.116194 454 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 4a625fcb95324466a12ab118bdb859f1 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:19.116494 454 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:19.116982 454 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:19.117230 454 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet a142157acbcd44adb6099b4428103ee5 (table default.Table_0870823e1f29409f9d61ebee0bbd060d [id=0a848dd5f6474712b2eba70add3743dc]). Expected version 3 got 0
I20251214 20:52:19.117551 454 catalog_manager.cc:5654] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:19.118033 454 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet cf04f91e79754569a51d468ab576b0dc (table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d]). Expected version 1 got 0
I20251214 20:52:19.118459 454 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:19.118808 454 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]). Expected version 1 got 0
I20251214 20:52:19.119004 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b7149a436a4cab80b39adccf8567f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.119664 991 leader_election.cc:304] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:19.120060 889 tablet_service.cc:1505] Processing CreateTablet for tablet f1eb4c3aa654480caaa69dbd5047365c (DEFAULT_TABLE table=default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.120510 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1eb4c3aa654480caaa69dbd5047365c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.121086 884 tablet_service.cc:1505] Processing CreateTablet for tablet 0d7c7b3f62764ff497f4d70b53eed7de (DEFAULT_TABLE table=default.Table_2def2ef868b34777b0585c4b95189baa [id=756e7314dcb5446f8e62baab11878de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.121690 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d7c7b3f62764ff497f4d70b53eed7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.123543 878 tablet_service.cc:1505] Processing CreateTablet for tablet ae948ffc81764692b92045f5befe0184 (DEFAULT_TABLE table=default.Table_488f7bbd18364d1e99b1f57aa7aba2c1 [id=6fc0adea2f63476d9603e83754155428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.123987 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae948ffc81764692b92045f5befe0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.126648 888 tablet_service.cc:1505] Processing CreateTablet for tablet d2a57409d4f54f17be792e6f3f17d7b7 (DEFAULT_TABLE table=default.Table_d7581ad6d39142f4a736c05d0e13811c [id=27d1b732a96a4ff3b0afc7a4fe59983b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.127151 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a57409d4f54f17be792e6f3f17d7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.128202 1898 tablet.cc:1722] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:19.129824 1856 ts_tablet_manager.cc:1929] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:19.130012 1856 log.cc:1199] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/c2ae9dc8ead8403cb7215af8778872f0
I20251214 20:52:19.130441 1856 ts_tablet_manager.cc:1950] T c2ae9dc8ead8403cb7215af8778872f0 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:19.131220 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 65693.5us
I20251214 20:52:19.131561 893 tablet_service.cc:1505] Processing CreateTablet for tablet e9f6da5d29294f08a86c96c6821dda10 (DEFAULT_TABLE table=default.Table_0dbf8224a9254c70a34ecf1cbcc79306 [id=9314ea76d87145ef9216ab81b340d426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.131989 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f6da5d29294f08a86c96c6821dda10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.132856 1799 leader_election.cc:290] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:19.132961 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bbb28259c1442ae94f4eff76c25c539" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:19.135329 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d4626d02ca548f3bf5a1a9142600bff" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:19.135532 883 tablet_service.cc:1505] Processing CreateTablet for tablet d01db63fd5d2474a9defdce0b98fc29d (DEFAULT_TABLE table=default.Table_26a82e34bb024bd89d6819c80056ba3b [id=fc160acc971e4f7d8290560ee8aab065]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.135833 1856 tablet_replica.cc:333] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:19.135914 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d01db63fd5d2474a9defdce0b98fc29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.136166 1055 raft_consensus.cc:3060] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:19.141809 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_00ec2fefc3d84a8b9f94bf9d194e2045"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:19.142505 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00ec2fefc3d84a8b9f94bf9d194e2045 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:19.144563 887 tablet_service.cc:1505] Processing CreateTablet for tablet 088c72a6f5014b6fbc6ee060b6b7a590 (DEFAULT_TABLE table=default.Table_ad73c92d80cf4c5d9f102d239903cb19 [id=a7cb6531a9e94ef5825c564dcaf8332a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.145965 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 088c72a6f5014b6fbc6ee060b6b7a590. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.146163 1897 raft_consensus.cc:2804] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:19.149374 1856 raft_consensus.cc:2243] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:19.149590 1856 raft_consensus.cc:2272] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:19.150425 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "60ae3db968f6412aa0c3952fafe65f88" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:19.152695 851 consensus_peers.cc:597] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:19.152690 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 49158.7us
I20251214 20:52:19.153007 1762 consensus_queue.cc:1048] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:19.156495 1856 ts_tablet_manager.cc:1916] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:19.156679 851 leader_election.cc:343] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.156821 851 leader_election.cc:304] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:19.156827 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet c2ae9dc8ead8403cb7215af8778872f0 (table default.Table_2342c4af4f9b420b9b7cbcf327ce1733 [id=976ca3fb3a164669abeff0d8eb137e41]) successfully deleted
I20251214 20:52:19.157243 1650 raft_consensus.cc:2749] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:19.157258 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3e5e220d083243eb8aa4328f366cd7ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e5e220d083243eb8aa4328f366cd7ff already in progress: creating tablet
I20251214 20:52:19.158293 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 088c72a6f5014b6fbc6ee060b6b7a590 (DEFAULT_TABLE table=default.Table_ad73c92d80cf4c5d9f102d239903cb19 [id=a7cb6531a9e94ef5825c564dcaf8332a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.158725 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 0d7c7b3f62764ff497f4d70b53eed7de (DEFAULT_TABLE table=default.Table_2def2ef868b34777b0585c4b95189baa [id=756e7314dcb5446f8e62baab11878de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.158886 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 088c72a6f5014b6fbc6ee060b6b7a590. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.159430 1897 raft_consensus.cc:697] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:19.159864 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d7c7b3f62764ff497f4d70b53eed7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.162779 991 leader_election.cc:304] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:19.177592 752 tablet_service.cc:1505] Processing CreateTablet for tablet 088c72a6f5014b6fbc6ee060b6b7a590 (DEFAULT_TABLE table=default.Table_ad73c92d80cf4c5d9f102d239903cb19 [id=a7cb6531a9e94ef5825c564dcaf8332a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.177675 1885 tablet.cc:1722] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:19.178035 754 tablet_service.cc:1505] Processing CreateTablet for tablet d01db63fd5d2474a9defdce0b98fc29d (DEFAULT_TABLE table=default.Table_26a82e34bb024bd89d6819c80056ba3b [id=fc160acc971e4f7d8290560ee8aab065]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.178130 713 leader_election.cc:304] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:19.178493 755 tablet_service.cc:1505] Processing CreateTablet for tablet 0d7c7b3f62764ff497f4d70b53eed7de (DEFAULT_TABLE table=default.Table_2def2ef868b34777b0585c4b95189baa [id=756e7314dcb5446f8e62baab11878de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.179203 756 tablet_service.cc:1505] Processing CreateTablet for tablet ae948ffc81764692b92045f5befe0184 (DEFAULT_TABLE table=default.Table_488f7bbd18364d1e99b1f57aa7aba2c1 [id=6fc0adea2f63476d9603e83754155428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.179723 742 tablet_service.cc:1505] Processing CreateTablet for tablet d2a57409d4f54f17be792e6f3f17d7b7 (DEFAULT_TABLE table=default.Table_d7581ad6d39142f4a736c05d0e13811c [id=27d1b732a96a4ff3b0afc7a4fe59983b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.180296 747 tablet_service.cc:1505] Processing CreateTablet for tablet f1eb4c3aa654480caaa69dbd5047365c (DEFAULT_TABLE table=default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.180908 1809 raft_consensus.cc:3060] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:19.181259 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b86f4adaf4540f48a4f42bc45e259e5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:19.181475 773 raft_consensus.cc:3060] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:19.181942 739 tablet_service.cc:1505] Processing CreateTablet for tablet e9f6da5d29294f08a86c96c6821dda10 (DEFAULT_TABLE table=default.Table_0dbf8224a9254c70a34ecf1cbcc79306 [id=9314ea76d87145ef9216ab81b340d426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.178161 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 088c72a6f5014b6fbc6ee060b6b7a590. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.181288 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8af99e0bd8774a678575847a9ec2e557" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:19.183634 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1eb4c3aa654480caaa69dbd5047365c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.184206 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f6da5d29294f08a86c96c6821dda10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.184739 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d01db63fd5d2474a9defdce0b98fc29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.185417 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a57409d4f54f17be792e6f3f17d7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.186270 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae948ffc81764692b92045f5befe0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.188730 1799 raft_consensus.cc:2804] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:19.189692 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d7c7b3f62764ff497f4d70b53eed7de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.189631 1889 tablet.cc:1722] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251214 20:52:19.197207 440 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 37a33cc9bb784f72956ac9f5493a0c3d consensus state has no leader
I20251214 20:52:19.197690 1809 raft_consensus.cc:2749] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:19.200735 773 raft_consensus.cc:2468] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:19.203572 1206 tablet_bootstrap.cc:492] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:19.206193 1206 tablet_bootstrap.cc:654] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.207401 1202 tablet_bootstrap.cc:492] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:19.207715 1202 ts_tablet_manager.cc:1403] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.212s user 0.029s sys 0.033s
I20251214 20:52:19.208828 1202 raft_consensus.cc:359] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.209210 1202 raft_consensus.cc:385] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.209712 1202 raft_consensus.cc:740] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.210078 1202 consensus_queue.cc:260] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.210837 1202 ts_tablet_manager.cc:1434] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:19.216190 1202 tablet_bootstrap.cc:492] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:19.224506 446 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 9d89ee602c524c96a21c0a235af07481
I20251214 20:52:19.225175 446 catalog_manager.cc:6724] ReplaceTablet: tablet 9d89ee602c524c96a21c0a235af07481 deleted and replaced by tablet 8e11723c042a4637839f1b13bed0935a
I20251214 20:52:19.229594 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_c2048db412744e2da280a49ec3d679b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:19.230365 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2048db412744e2da280a49ec3d679b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:19.239518 1885 tablet.cc:1722] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:19.241014 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d89ee602c524c96a21c0a235af07481 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e11723c042a4637839f1b13bed0935a) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.242923 1650 raft_consensus.cc:493] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.243165 1650 raft_consensus.cc:515] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.243825 1650 leader_election.cc:290] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:19.246956 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a8662a634a4befbd2d6251000d2059" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:19.248381 848 leader_election.cc:343] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:19.248922 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 9d89ee602c524c96a21c0a235af07481 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d89ee602c524c96a21c0a235af07481 already in progress: creating tablet
I20251214 20:52:19.253676 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ae7fe3d1672c46baa454f28810e589af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:19.254732 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae7fe3d1672c46baa454f28810e589af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:19.260406 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d89ee602c524c96a21c0a235af07481 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e11723c042a4637839f1b13bed0935a) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:19.261243 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 9d89ee602c524c96a21c0a235af07481 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d89ee602c524c96a21c0a235af07481 already in progress: creating tablet
I20251214 20:52:19.263024 1206 tablet_bootstrap.cc:492] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:19.263312 1206 ts_tablet_manager.cc:1403] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.162s user 0.001s sys 0.003s
I20251214 20:52:19.264603 991 leader_election.cc:304] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:19.264582 1885 tablet.cc:1722] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251214 20:52:19.264966 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 100549us
I20251214 20:52:19.266163 1799 raft_consensus.cc:697] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:19.266144 1206 raft_consensus.cc:359] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.266331 1206 raft_consensus.cc:385] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.266408 1206 raft_consensus.cc:740] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.267436 1908 tablet.cc:1722] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:19.267489 1033 tablet_service.cc:1505] Processing CreateTablet for tablet ae948ffc81764692b92045f5befe0184 (DEFAULT_TABLE table=default.Table_488f7bbd18364d1e99b1f57aa7aba2c1 [id=6fc0adea2f63476d9603e83754155428]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.267908 1206 consensus_queue.cc:260] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.270220 1799 consensus_queue.cc:237] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.270879 1206 ts_tablet_manager.cc:1434] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251214 20:52:19.264478 1897 consensus_queue.cc:237] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.274029 1035 tablet_service.cc:1505] Processing CreateTablet for tablet f1eb4c3aa654480caaa69dbd5047365c (DEFAULT_TABLE table=default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.275339 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f4760585baf4b4ab3b9b94b5a748a61" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:19.274178 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d89ee602c524c96a21c0a235af07481 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e11723c042a4637839f1b13bed0935a) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.274066 1029 tablet_service.cc:1505] Processing CreateTablet for tablet d01db63fd5d2474a9defdce0b98fc29d (DEFAULT_TABLE table=default.Table_26a82e34bb024bd89d6819c80056ba3b [id=fc160acc971e4f7d8290560ee8aab065]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.275115 1809 consensus_queue.cc:1048] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 20:52:19.275923 713 leader_election.cc:304] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:19.276991 1809 raft_consensus.cc:3060] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:19.277163 991 consensus_peers.cc:597] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63497e77baeb4d328be2421b5732a28a. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:19.278136 851 leader_election.cc:343] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.278280 851 leader_election.cc:304] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:19.278872 1762 raft_consensus.cc:2749] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:19.279932 1809 raft_consensus.cc:2749] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:19.286751 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8076bc8b31ba411faf089f314e6ed2ef" } modify_external_catalogs: true
I20251214 20:52:19.287478 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8076bc8b31ba411faf089f314e6ed2ef" } modify_external_catalogs: true
I20251214 20:52:19.295864 1667 consensus_queue.cc:1048] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20251214 20:52:19.296540 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae948ffc81764692b92045f5befe0184. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.298877 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1eb4c3aa654480caaa69dbd5047365c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.302971 1203 tablet_bootstrap.cc:492] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:19.271519 1206 tablet_bootstrap.cc:492] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:19.311211 1206 tablet_bootstrap.cc:654] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.311655 1664 raft_consensus.cc:2804] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:19.312431 1202 tablet_bootstrap.cc:654] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.313613 1804 consensus_queue.cc:1048] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20251214 20:52:19.313937 1034 tablet_service.cc:1505] Processing CreateTablet for tablet d2a57409d4f54f17be792e6f3f17d7b7 (DEFAULT_TABLE table=default.Table_d7581ad6d39142f4a736c05d0e13811c [id=27d1b732a96a4ff3b0afc7a4fe59983b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.314404 1025 tablet_service.cc:1505] Processing CreateTablet for tablet e9f6da5d29294f08a86c96c6821dda10 (DEFAULT_TABLE table=default.Table_0dbf8224a9254c70a34ecf1cbcc79306 [id=9314ea76d87145ef9216ab81b340d426]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.325733 1664 raft_consensus.cc:697] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:19.334739 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:19.339499 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]). Expected version 1 got 0
I20251214 20:52:19.341159 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a8662a634a4befbd2d6251000d2059" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:19.341747 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 4a625fcb95324466a12ab118bdb859f1 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
W20251214 20:52:19.341789 851 leader_election.cc:343] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.341950 851 leader_election.cc:304] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:19.342227 1650 raft_consensus.cc:2749] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:19.345469 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 1 got 0
I20251214 20:52:19.347856 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d01db63fd5d2474a9defdce0b98fc29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.348255 1202 tablet_bootstrap.cc:492] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:19.348455 1202 ts_tablet_manager.cc:1403] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.132s user 0.002s sys 0.002s
I20251214 20:52:19.348518 1206 tablet_bootstrap.cc:492] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:19.348695 1206 ts_tablet_manager.cc:1403] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.000s
I20251214 20:52:19.349422 1206 raft_consensus.cc:359] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:19.349617 1206 raft_consensus.cc:385] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.349680 1206 raft_consensus.cc:740] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.349892 1206 consensus_queue.cc:260] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:19.349439 1202 raft_consensus.cc:359] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:19.353544 1202 raft_consensus.cc:385] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.353741 1202 raft_consensus.cc:740] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.354192 1202 consensus_queue.cc:260] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:19.354992 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet a142157acbcd44adb6099b4428103ee5 (table default.Table_0870823e1f29409f9d61ebee0bbd060d [id=0a848dd5f6474712b2eba70add3743dc]). Expected version 3 got 0
I20251214 20:52:19.362972 1664 consensus_queue.cc:237] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.363335 1206 ts_tablet_manager.cc:1434] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251214 20:52:19.364512 1203 ts_tablet_manager.cc:1403] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.483s user 0.017s sys 0.000s
I20251214 20:52:19.365657 1202 ts_tablet_manager.cc:1434] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251214 20:52:19.365932 1892 tablet.cc:1722] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251214 20:52:19.367523 1206 tablet_bootstrap.cc:492] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:19.370404 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:19.370617 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]). Expected version 1 got 0
I20251214 20:52:19.370838 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:19.371367 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 144dee1a6d7344f2b79da6e79e67599e (table default.Table_31645b7a21d345ef9bc06c0ba9b2c5fa [id=8cd62da3d57c4be4b5f10ace49d2e148]). Expected version 4 got 0
I20251214 20:52:19.371704 1206 tablet_bootstrap.cc:654] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.378074 1202 tablet_bootstrap.cc:492] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:19.379916 1202 tablet_bootstrap.cc:654] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.385075 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1]). Expected version 1 got 0
I20251214 20:52:19.385383 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:19.385673 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:19.385880 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 65e79db74fa046698507c83416ee1669 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]). Expected version 1 got 0
I20251214 20:52:19.386083 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2accf8a160584c82bb695a52e3f2b05c (table default.Table_b4334a1d99a54c2eabe2fb909563313c [id=d31b208e2e284575ad429878129a28f5]). Expected version 2 got 0
I20251214 20:52:19.386576 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:19.386972 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:19.392159 1898 tablet.cc:1722] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:19.392840 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a57409d4f54f17be792e6f3f17d7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.394517 1203 raft_consensus.cc:359] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.394750 1203 raft_consensus.cc:385] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.394824 1203 raft_consensus.cc:740] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.395056 1203 consensus_queue.cc:260] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.395596 1203 ts_tablet_manager.cc:1434] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20251214 20:52:19.395962 1203 tablet_bootstrap.cc:492] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:19.396867 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 96990us
I20251214 20:52:19.401345 1202 tablet_bootstrap.cc:492] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:19.402101 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:19.402209 1889 tablet.cc:1722] T 4a625fcb95324466a12ab118bdb859f1 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:19.404536 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f6da5d29294f08a86c96c6821dda10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.403815 1903 tablet.cc:1722] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:19.411329 1856 ts_tablet_manager.cc:1929] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:19.411512 1856 log.cc:1199] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/9c0f49e5d78240a296e47eccb3218e4f
I20251214 20:52:19.411882 1856 ts_tablet_manager.cc:1950] T 9c0f49e5d78240a296e47eccb3218e4f P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:19.412823 1856 tablet_replica.cc:333] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:19.413048 1856 raft_consensus.cc:2243] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:19.413216 1856 raft_consensus.cc:2272] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:19.413713 1856 ts_tablet_manager.cc:1916] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:19.416188 1908 tablet.cc:1722] T 4a625fcb95324466a12ab118bdb859f1 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:19.418303 1206 tablet_bootstrap.cc:492] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:19.418490 1206 ts_tablet_manager.cc:1403] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.005s
I20251214 20:52:19.419374 1206 raft_consensus.cc:359] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.419441 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b86f4adaf4540f48a4f42bc45e259e5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:19.419586 1206 raft_consensus.cc:385] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.419659 912 raft_consensus.cc:3060] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:19.419678 1206 raft_consensus.cc:740] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.419909 1206 consensus_queue.cc:260] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.420460 1206 ts_tablet_manager.cc:1434] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:19.420919 1202 ts_tablet_manager.cc:1403] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.004s
I20251214 20:52:19.421919 1202 raft_consensus.cc:359] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.422179 1202 raft_consensus.cc:385] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.422256 1202 raft_consensus.cc:740] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.423045 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 0c973e0111804b8b869068bdcc5869bf (table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f]). Expected version 1 got 0
I20251214 20:52:19.425288 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet e16fe751f10f4616a00d0af811998729 (table default.Table_7923a9cba4ce4b27a12617e394ffbf2e [id=8fd9be1206a74aa1921e0dade8be3774]). Expected version 1 got 0
I20251214 20:52:19.426560 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 9c0f49e5d78240a296e47eccb3218e4f (table default.Table_ec65ee1eda49435895dab50f78b4ee15 [id=8a0182244a254993998d4ea00d03cd9d]) successfully deleted
I20251214 20:52:19.426667 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:19.427212 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:19.427549 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:19.429090 1206 tablet_bootstrap.cc:492] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:19.429380 912 raft_consensus.cc:2468] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:19.433243 1206 tablet_bootstrap.cc:654] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:19.435009 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 167668us
W20251214 20:52:19.442162 989 consensus_peers.cc:597] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:19.442447 989 leader_election.cc:343] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.442698 989 leader_election.cc:304] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:19.443480 1595 raft_consensus.cc:2804] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:19.443710 1595 raft_consensus.cc:697] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:19.443980 1799 consensus_queue.cc:1048] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251214 20:52:19.444015 1595 consensus_queue.cc:237] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.444034 1202 consensus_queue.cc:260] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.444906 1799 consensus_queue.cc:1048] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:19.445371 1799 raft_consensus.cc:493] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.445521 1799 raft_consensus.cc:515] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.445667 1903 tablet.cc:1722] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:19.446151 1799 leader_election.cc:290] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:19.450518 768 raft_consensus.cc:3060] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:19.452863 1202 ts_tablet_manager.cc:1434] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20251214 20:52:19.459451 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet cf04f91e79754569a51d468ab576b0dc (table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d]). Expected version 1 got 0
I20251214 20:52:19.459867 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 1 got 0
I20251214 20:52:19.461580 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 2 got 1
I20251214 20:52:19.461869 1202 tablet_bootstrap.cc:492] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:19.461902 1206 tablet_bootstrap.cc:492] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:19.462625 1206 ts_tablet_manager.cc:1403] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.008s
I20251214 20:52:19.463907 1202 tablet_bootstrap.cc:654] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.463833 1206 raft_consensus.cc:359] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.464278 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]). Expected version 3 got 0
I20251214 20:52:19.464700 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:19.464893 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_55fe32e16f364e58afdd57c80a8966ae [id=6157817e96d642d09f9941ad7f7e82ee]). Expected version 1 got 0
I20251214 20:52:19.465026 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:19.465309 1206 raft_consensus.cc:385] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.465577 1206 raft_consensus.cc:740] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.466022 1206 consensus_queue.cc:260] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.467343 1206 ts_tablet_manager.cc:1434] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:19.468235 1206 tablet_bootstrap.cc:492] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:19.471563 1206 tablet_bootstrap.cc:654] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.485661 1915 consensus_queue.cc:1048] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20251214 20:52:19.492982 1203 tablet_bootstrap.cc:654] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.493064 1882 consensus_queue.cc:1048] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251214 20:52:19.493822 1897 consensus_queue.cc:1048] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20251214 20:52:19.499323 1800 consensus_queue.cc:1048] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20251214 20:52:19.501497 1804 consensus_queue.cc:1048] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20251214 20:52:19.521704 1856 ts_tablet_manager.cc:1929] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:19.528690 1856 log.cc:1199] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/8af99e0bd8774a678575847a9ec2e557
I20251214 20:52:19.528645 1804 consensus_queue.cc:1048] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:19.529299 1856 ts_tablet_manager.cc:1950] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:19.528645 1898 tablet.cc:1722] T cf04f91e79754569a51d468ab576b0dc P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:19.529973 1856 tablet_replica.cc:333] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:19.530287 1856 raft_consensus.cc:2243] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:19.530593 1856 pending_rounds.cc:70] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:19.530742 1856 pending_rounds.cc:77] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493723268583424 op_type: NO_OP noop_request { }
I20251214 20:52:19.530874 1856 raft_consensus.cc:2889] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:19.531068 1856 raft_consensus.cc:2272] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:19.531708 1856 ts_tablet_manager.cc:1916] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:19.532353 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 8af99e0bd8774a678575847a9ec2e557 (table default.Table_ade67d7e10e04c6fb0cfc134947a559d [id=7986a565ce914875bf3e24c1cc2c288a]) successfully deleted
I20251214 20:52:19.546295 1804 consensus_queue.cc:1048] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251214 20:52:19.477056 902 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34eb8fa3e34427495fa5985b7ba81bf" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:19.480167 1889 tablet.cc:1722] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:19.571907 902 raft_consensus.cc:2468] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:19.481420 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8af99e0bd8774a678575847a9ec2e557" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:19.572624 911 raft_consensus.cc:3060] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:19.579918 1889 tablet.cc:1722] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:19.581938 911 raft_consensus.cc:2468] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:19.494153 1932 tablet.cc:1722] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:19.584185 1889 tablet.cc:1722] T cf04f91e79754569a51d468ab576b0dc P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:19.586079 1650 raft_consensus.cc:493] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.586349 1650 raft_consensus.cc:515] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.587774 1650 leader_election.cc:290] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:19.587904 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9b7149a436a4cab80b39adccf8567f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:19.588693 848 leader_election.cc:343] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.590713 1203 tablet_bootstrap.cc:492] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:19.591177 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9b7149a436a4cab80b39adccf8567f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:19.591527 989 consensus_peers.cc:597] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:19.591903 1595 consensus_queue.cc:1048] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:19.592855 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34eb8fa3e34427495fa5985b7ba81bf" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:19.593878 774 raft_consensus.cc:3060] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:19.600342 851 leader_election.cc:343] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.600587 851 leader_election.cc:304] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:19.604347 1650 raft_consensus.cc:2749] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:19.606398 1908 tablet.cc:1722] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251214 20:52:19.614826 989 consensus_peers.cc:597] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:19.616535 1203 ts_tablet_manager.cc:1403] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.195s user 0.004s sys 0.016s
I20251214 20:52:19.617657 1203 raft_consensus.cc:359] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.617944 1203 raft_consensus.cc:385] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.618068 1203 raft_consensus.cc:740] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.618412 1203 consensus_queue.cc:260] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.619766 1203 ts_tablet_manager.cc:1434] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:19.620705 1203 tablet_bootstrap.cc:492] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:19.622375 1203 tablet_bootstrap.cc:654] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.622501 991 leader_election.cc:304] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:19.641538 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 64806.4us
W20251214 20:52:19.642486 989 leader_election.cc:343] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:19.643249 989 consensus_peers.cc:597] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:19.643632 1595 raft_consensus.cc:2804] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:19.643725 1595 raft_consensus.cc:493] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:19.643836 1595 raft_consensus.cc:3060] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:19.646842 1595 raft_consensus.cc:515] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.647570 1667 consensus_queue.cc:1048] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:19.647735 1928 consensus_queue.cc:1048] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:19.648408 1595 leader_election.cc:290] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:19.638934 1898 tablet.cc:1722] T 4a625fcb95324466a12ab118bdb859f1 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:19.650689 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34eb8fa3e34427495fa5985b7ba81bf" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:19.655046 1882 consensus_queue.cc:1048] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:19.655817 989 leader_election.cc:343] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.677512 1856 ts_tablet_manager.cc:1929] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:19.677806 1856 log.cc:1199] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/52daf1204bf142e7b82852bdec6582fb
I20251214 20:52:19.678277 1856 ts_tablet_manager.cc:1950] T 52daf1204bf142e7b82852bdec6582fb P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:19.679623 442 consensus_peers.cc:597] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a -> Peer e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819): Couldn't send request to peer e6713a20a82b434f8a8bc122113e26ad. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251214 20:52:19.679759 1856 tablet_replica.cc:333] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:19.680441 1856 raft_consensus.cc:2243] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:19.681533 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 52daf1204bf142e7b82852bdec6582fb (table default.Table_5fbbdeeea9a14e17bf6f9d0cc4a56d98 [id=a0e13ae34009499d91b4461520c292c1]) successfully deleted
I20251214 20:52:19.683521 1762 raft_consensus.cc:493] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.683537 754 tablet_service.cc:1505] Processing CreateTablet for tablet 9f04336d2c84439b9003df19d3541273 (DEFAULT_TABLE table=default.Table_7041fb3e6d4040f08fd9a554a30e315d [id=b6e9928c0c2845fdb544b4b449aeb113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.683852 1762 raft_consensus.cc:515] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:19.684111 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f04336d2c84439b9003df19d3541273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.685108 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3962c33a294fa98f8bd485268aee71" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:19.685719 848 leader_election.cc:343] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.691572 1762 leader_election.cc:290] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:19.700706 1202 tablet_bootstrap.cc:492] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:19.700956 1202 ts_tablet_manager.cc:1403] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.240s user 0.003s sys 0.003s
I20251214 20:52:19.682178 991 leader_election.cc:304] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:19.718932 1202 raft_consensus.cc:359] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.720583 1206 tablet_bootstrap.cc:492] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:19.720853 1206 ts_tablet_manager.cc:1403] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.253s user 0.048s sys 0.082s
I20251214 20:52:19.721774 1206 raft_consensus.cc:359] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.722011 1206 raft_consensus.cc:385] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.722151 1206 raft_consensus.cc:740] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.722376 887 tablet_service.cc:1505] Processing CreateTablet for tablet 9f04336d2c84439b9003df19d3541273 (DEFAULT_TABLE table=default.Table_7041fb3e6d4040f08fd9a554a30e315d [id=b6e9928c0c2845fdb544b4b449aeb113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.722479 1206 consensus_queue.cc:260] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.722827 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f04336d2c84439b9003df19d3541273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.723037 1206 ts_tablet_manager.cc:1434] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:19.723544 1206 tablet_bootstrap.cc:492] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:19.725766 1206 tablet_bootstrap.cc:654] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.729457 1202 raft_consensus.cc:385] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.729650 1202 raft_consensus.cc:740] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.729532 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3962c33a294fa98f8bd485268aee71" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:19.730087 1202 consensus_queue.cc:260] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:19.730367 851 leader_election.cc:343] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.731226 851 leader_election.cc:304] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:19.732120 1762 raft_consensus.cc:2749] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:19.732896 1202 ts_tablet_manager.cc:1434] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20251214 20:52:19.734555 1206 tablet_bootstrap.cc:492] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:19.734762 1206 ts_tablet_manager.cc:1403] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20251214 20:52:19.735347 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 9f04336d2c84439b9003df19d3541273 (DEFAULT_TABLE table=default.Table_7041fb3e6d4040f08fd9a554a30e315d [id=b6e9928c0c2845fdb544b4b449aeb113]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.735637 1206 raft_consensus.cc:359] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.735836 1206 raft_consensus.cc:385] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.735900 1206 raft_consensus.cc:740] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.735935 1856 raft_consensus.cc:2272] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:19.732191 991 raft_consensus.cc:2686] T 8af99e0bd8774a678575847a9ec2e557 P 63848a9fb78a4013a6f1657621272d75: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251214 20:52:19.736166 1206 consensus_queue.cc:260] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.736858 1206 ts_tablet_manager.cc:1434] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:19.737205 1206 tablet_bootstrap.cc:492] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:19.739332 1206 tablet_bootstrap.cc:654] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.739650 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f04336d2c84439b9003df19d3541273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.740046 446 catalog_manager.cc:5654] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:19.740717 754 tablet_service.cc:1505] Processing CreateTablet for tablet 292a28b12df246a8bc855d81c530746e (DEFAULT_TABLE table=default.Table_9fa9404adcd247179c0695d6a85e87ea [id=d4ffef9acf9745b4949f47355f6d6e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.741971 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292a28b12df246a8bc855d81c530746e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.743359 1202 tablet_bootstrap.cc:492] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:19.744654 887 tablet_service.cc:1505] Processing CreateTablet for tablet 292a28b12df246a8bc855d81c530746e (DEFAULT_TABLE table=default.Table_9fa9404adcd247179c0695d6a85e87ea [id=d4ffef9acf9745b4949f47355f6d6e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.745095 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292a28b12df246a8bc855d81c530746e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.751081 1202 tablet_bootstrap.cc:654] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.755625 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 292a28b12df246a8bc855d81c530746e (DEFAULT_TABLE table=default.Table_9fa9404adcd247179c0695d6a85e87ea [id=d4ffef9acf9745b4949f47355f6d6e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.756148 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292a28b12df246a8bc855d81c530746e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.757045 1856 ts_tablet_manager.cc:1916] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:19.754220 446 catalog_manager.cc:5654] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:19.758949 739 tablet_service.cc:1505] Processing CreateTablet for tablet 8deabfdbd0674b2884ee5f82fa5e332f (DEFAULT_TABLE table=default.Table_3bc5f340978c453190d06f9e7a87f65c [id=87835c51994146ffa7f67ece6f50ff40]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:19.759220 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 66701us
I20251214 20:52:19.759514 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8deabfdbd0674b2884ee5f82fa5e332f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.759025 446 catalog_manager.cc:5654] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:19.760694 878 tablet_service.cc:1505] Processing CreateTablet for tablet 8deabfdbd0674b2884ee5f82fa5e332f (DEFAULT_TABLE table=default.Table_3bc5f340978c453190d06f9e7a87f65c [id=87835c51994146ffa7f67ece6f50ff40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.767444 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8deabfdbd0674b2884ee5f82fa5e332f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.769721 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 8deabfdbd0674b2884ee5f82fa5e332f (DEFAULT_TABLE table=default.Table_3bc5f340978c453190d06f9e7a87f65c [id=87835c51994146ffa7f67ece6f50ff40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.770169 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8deabfdbd0674b2884ee5f82fa5e332f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.771543 1800 raft_consensus.cc:493] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.772153 1800 raft_consensus.cc:515] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:19.776670 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]). Expected version 1 got 0
I20251214 20:52:19.777611 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 4a625fcb95324466a12ab118bdb859f1 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]). Expected version 1 got 0
I20251214 20:52:19.778755 878 tablet_service.cc:1505] Processing CreateTablet for tablet 1d7aa508b11944aa8e268741c1998a9e (DEFAULT_TABLE table=default.Table_989a947ef29f422cbcc7fd84b5cac678 [id=cabd478801a54c1786a1df14a3c046af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.779250 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7aa508b11944aa8e268741c1998a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.779721 754 tablet_service.cc:1505] Processing CreateTablet for tablet 1d7aa508b11944aa8e268741c1998a9e (DEFAULT_TABLE table=default.Table_989a947ef29f422cbcc7fd84b5cac678 [id=cabd478801a54c1786a1df14a3c046af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.780758 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7aa508b11944aa8e268741c1998a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.783699 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 1 got 0
I20251214 20:52:19.784936 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:19.786101 739 tablet_service.cc:1505] Processing CreateTablet for tablet 79e621f29c01448399d22615284fccd4 (DEFAULT_TABLE table=default.Table_f1a942f776f74d61922b38c19e43f5df [id=6094596f6c064c5bb1e20aa2b8767b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.786761 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e621f29c01448399d22615284fccd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.787705 1206 tablet_bootstrap.cc:492] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:19.787899 1206 ts_tablet_manager.cc:1403] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.015s
I20251214 20:52:19.788489 887 tablet_service.cc:1505] Processing CreateTablet for tablet 79e621f29c01448399d22615284fccd4 (DEFAULT_TABLE table=default.Table_f1a942f776f74d61922b38c19e43f5df [id=6094596f6c064c5bb1e20aa2b8767b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.789031 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e621f29c01448399d22615284fccd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.788964 1206 raft_consensus.cc:359] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.789202 1206 raft_consensus.cc:385] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.789314 1206 raft_consensus.cc:740] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.789566 1206 consensus_queue.cc:260] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.790176 1206 ts_tablet_manager.cc:1434] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:19.790506 1206 tablet_bootstrap.cc:492] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:19.794348 1206 tablet_bootstrap.cc:654] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.795497 1800 leader_election.cc:290] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:19.796358 446 catalog_manager.cc:5654] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:19.796913 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet a142157acbcd44adb6099b4428103ee5 (table default.Table_0870823e1f29409f9d61ebee0bbd060d [id=0a848dd5f6474712b2eba70add3743dc]). Expected version 3 got 0
I20251214 20:52:19.797178 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:19.797425 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]). Expected version 1 got 0
I20251214 20:52:19.797737 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 144dee1a6d7344f2b79da6e79e67599e (table default.Table_31645b7a21d345ef9bc06c0ba9b2c5fa [id=8cd62da3d57c4be4b5f10ace49d2e148]). Expected version 4 got 0
I20251214 20:52:19.797876 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1]). Expected version 1 got 0
I20251214 20:52:19.798098 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:19.798296 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:19.798441 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:19.798604 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2accf8a160584c82bb695a52e3f2b05c (table default.Table_b4334a1d99a54c2eabe2fb909563313c [id=d31b208e2e284575ad429878129a28f5]). Expected version 2 got 0
I20251214 20:52:19.798858 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:19.800230 1206 tablet_bootstrap.cc:492] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:19.800472 1206 ts_tablet_manager.cc:1403] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20251214 20:52:19.801416 1206 raft_consensus.cc:359] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.801694 1206 raft_consensus.cc:385] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.801806 1206 raft_consensus.cc:740] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.802214 1206 consensus_queue.cc:260] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.803314 1206 ts_tablet_manager.cc:1434] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:19.804262 1206 tablet_bootstrap.cc:492] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:19.808182 884 tablet_service.cc:1505] Processing CreateTablet for tablet 6eca34627bbc4a3f9915be72f3128370 (DEFAULT_TABLE table=default.Table_a91064ac3fb94282a02fbc867fdd581c [id=6b97974ca1764265aa6c291f8439da8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.810554 1202 tablet_bootstrap.cc:492] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:19.810710 1202 ts_tablet_manager.cc:1403] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20251214 20:52:19.811343 1202 raft_consensus.cc:359] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.811483 878 tablet_service.cc:1505] Processing CreateTablet for tablet dfe50a6219134431b0a025317932609b (DEFAULT_TABLE table=default.Table_b5f628ed069a4b19996c9b579d7678bc [id=a03c04d94997449bac7447ed634577d4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:19.811496 989 leader_election.cc:343] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 569a02142c71427d91868b02695ae12b
I20251214 20:52:19.811733 1202 raft_consensus.cc:385] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.811879 1202 raft_consensus.cc:740] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.812199 1202 consensus_queue.cc:260] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.812496 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 79e621f29c01448399d22615284fccd4 (DEFAULT_TABLE table=default.Table_f1a942f776f74d61922b38c19e43f5df [id=6094596f6c064c5bb1e20aa2b8767b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.816042 1035 tablet_service.cc:1505] Processing CreateTablet for tablet dfe50a6219134431b0a025317932609b (DEFAULT_TABLE table=default.Table_b5f628ed069a4b19996c9b579d7678bc [id=a03c04d94997449bac7447ed634577d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.816195 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:19.816377 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 6eca34627bbc4a3f9915be72f3128370 (DEFAULT_TABLE table=default.Table_a91064ac3fb94282a02fbc867fdd581c [id=6b97974ca1764265aa6c291f8439da8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.816587 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 0c973e0111804b8b869068bdcc5869bf (table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f]). Expected version 1 got 0
I20251214 20:52:19.816854 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e621f29c01448399d22615284fccd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.816851 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e16fe751f10f4616a00d0af811998729 (table default.Table_7923a9cba4ce4b27a12617e394ffbf2e [id=8fd9be1206a74aa1921e0dade8be3774]). Expected version 1 got 0
I20251214 20:52:19.817009 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:19.817239 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:19.817804 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6eca34627bbc4a3f9915be72f3128370. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.818261 888 tablet_service.cc:1505] Processing CreateTablet for tablet 80098b7c24824d129b8f25daf2115460 (DEFAULT_TABLE table=default.Table_8ca3b0a5a22a425b866b0b2565f80fac [id=2a055be9028342be97f5c98f03f5a137]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.818605 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80098b7c24824d129b8f25daf2115460. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:19.823439 991 consensus_peers.cc:597] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b4f582d8165439d86d9228c146dc41a. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:19.824940 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34eb8fa3e34427495fa5985b7ba81bf" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:19.825227 912 raft_consensus.cc:3060] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:19.826489 1202 ts_tablet_manager.cc:1434] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251214 20:52:19.827351 1202 tablet_bootstrap.cc:492] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:19.829094 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfe50a6219134431b0a025317932609b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.830277 912 raft_consensus.cc:2468] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:19.830713 888 tablet_service.cc:1505] Processing CreateTablet for tablet c8e33ba1149a44c294885d76f676160a (DEFAULT_TABLE table=default.Table_3f7396ac07d649fa8d342ae05765da80 [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.833724 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e33ba1149a44c294885d76f676160a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.834249 1202 tablet_bootstrap.cc:654] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.839529 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfe50a6219134431b0a025317932609b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.842784 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 1d7aa508b11944aa8e268741c1998a9e (DEFAULT_TABLE table=default.Table_989a947ef29f422cbcc7fd84b5cac678 [id=cabd478801a54c1786a1df14a3c046af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.848906 1664 raft_consensus.cc:493] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.817517 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 65e79db74fa046698507c83416ee1669 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]). Expected version 1 got 0
I20251214 20:52:19.851296 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:19.851293 888 tablet_service.cc:1505] Processing CreateTablet for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (DEFAULT_TABLE table=default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.851534 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:19.851789 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.855729 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet cf04f91e79754569a51d468ab576b0dc (table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d]). Expected version 1 got 0
I20251214 20:52:19.857757 1964 raft_consensus.cc:493] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.857946 1964 raft_consensus.cc:515] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.858630 1964 leader_election.cc:290] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:19.859733 712 leader_election.cc:343] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: bd8226b431c6456885e928880938969d
I20251214 20:52:19.859753 887 tablet_service.cc:1505] Processing CreateTablet for tablet a07d94d5a431488489ee9fbcc113a9d7 (DEFAULT_TABLE table=default.Table_493f60b9d8024509856ed934d379f829 [id=a40a329d98674364aa3794d3a07fdeb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.860155 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07d94d5a431488489ee9fbcc113a9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.863044 1664 raft_consensus.cc:515] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.863619 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 2 got 1
I20251214 20:52:19.863904 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]). Expected version 3 got 0
I20251214 20:52:19.864183 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:19.864331 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_55fe32e16f364e58afdd57c80a8966ae [id=6157817e96d642d09f9941ad7f7e82ee]). Expected version 1 got 0
I20251214 20:52:19.864537 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:19.867328 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet fe4086fc7a4f4db0b7541f06c9833aab
I20251214 20:52:19.868180 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_816a237b5485489c9dc522f872701944"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:19.868708 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_816a237b5485489c9dc522f872701944 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:19.869530 991 leader_election.cc:304] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:19.870342 1804 raft_consensus.cc:2804] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
W20251214 20:52:19.870666 991 leader_election.cc:343] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 569a02142c71427d91868b02695ae12b
I20251214 20:52:19.871176 1664 leader_election.cc:290] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:19.871212 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6eca34627bbc4a3f9915be72f3128370. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.873045 1804 raft_consensus.cc:697] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:19.873431 1804 consensus_queue.cc:237] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.872519 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 80098b7c24824d129b8f25daf2115460 (DEFAULT_TABLE table=default.Table_8ca3b0a5a22a425b866b0b2565f80fac [id=2a055be9028342be97f5c98f03f5a137]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.878263 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9ad469dea994477983bfe62eef37e334" } modify_external_catalogs: true
I20251214 20:52:19.878373 1026 tablet_service.cc:1505] Processing CreateTablet for tablet a07d94d5a431488489ee9fbcc113a9d7 (DEFAULT_TABLE table=default.Table_493f60b9d8024509856ed934d379f829 [id=a40a329d98674364aa3794d3a07fdeb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.878532 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9ad469dea994477983bfe62eef37e334" } modify_external_catalogs: true
I20251214 20:52:19.878383 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (DEFAULT_TABLE table=default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.878400 1024 tablet_service.cc:1505] Processing CreateTablet for tablet c8e33ba1149a44c294885d76f676160a (DEFAULT_TABLE table=default.Table_3f7396ac07d649fa8d342ae05765da80 [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.879376 884 tablet_service.cc:1552] Processing DeleteTablet for tablet fe4086fc7a4f4db0b7541f06c9833aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.881192 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 727e7b61ae3942098712720d344c5310 (DEFAULT_TABLE table=default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.881377 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet fe4086fc7a4f4db0b7541f06c9833aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.882576 1202 tablet_bootstrap.cc:492] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:19.882800 1202 ts_tablet_manager.cc:1403] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s
I20251214 20:52:19.883381 823 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:19.883749 1202 raft_consensus.cc:359] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.884114 1202 raft_consensus.cc:385] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.884246 1202 raft_consensus.cc:740] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.884554 1202 consensus_queue.cc:260] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.885509 1202 ts_tablet_manager.cc:1434] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:19.896826 1202 tablet_bootstrap.cc:492] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:19.899353 1962 raft_consensus.cc:493] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.899616 1202 tablet_bootstrap.cc:654] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.899617 1962 raft_consensus.cc:515] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.900597 1962 leader_election.cc:290] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:19.901003 888 tablet_service.cc:1505] Processing CreateTablet for tablet 727e7b61ae3942098712720d344c5310 (DEFAULT_TABLE table=default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.901681 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 727e7b61ae3942098712720d344c5310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.902433 991 leader_election.cc:304] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:19.902515 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d7aa508b11944aa8e268741c1998a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.902902 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 727e7b61ae3942098712720d344c5310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.901692 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b9ca42a5ce4572aebe2a24f4758b12" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:19.903270 912 raft_consensus.cc:2393] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:19.903610 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a8662a634a4befbd2d6251000d2059" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:19.903879 910 raft_consensus.cc:2468] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:19.909595 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07d94d5a431488489ee9fbcc113a9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.909560 1928 raft_consensus.cc:2749] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:19.912597 448 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 98b7cac3930a4a9eb16ac413c8f1e092
W20251214 20:52:19.912740 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e11723c042a4637839f1b13bed0935a has no consensus state
I20251214 20:52:19.912910 450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 86) took 1056 ms. Trace:
I20251214 20:52:19.913206 448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 83) took 1120 ms. Trace:
I20251214 20:52:19.913417 452 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 7dd657f3baff40bb8cb7f4648367f7a1
I20251214 20:52:19.913861 452 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 82) took 1147 ms. Trace:
I20251214 20:52:19.914045 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e33ba1149a44c294885d76f676160a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.913956 452 rpcz_store.cc:276] 1214 20:52:18.766416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.766467 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:18.767516 (+ 1049us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:18.767720 (+ 204us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:18.767807 (+ 87us) catalog_manager.cc:2824] Locking tablets
1214 20:52:19.097502 (+329695us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:19.227736 (+130234us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:19.227817 (+ 81us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:19.913377 (+685560us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:19.913805 (+ 428us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:19.100382 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:19.100545 (+ 163us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:19.100551 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:19.100554 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:19.100694 (+ 140us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:19.100708 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:19.100715 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:19.100851 (+ 136us) tablet.cc:699] Row locks acquired
1214 20:52:19.100854 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:19.100895 (+ 41us) write_op.cc:270] Start()
1214 20:52:19.100935 (+ 40us) write_op.cc:276] Timestamp: P: 1765745539100864 usec, L: 0
1214 20:52:19.100939 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:19.108455 (+ 7516us) log.cc:844] Serialized 725 byte log entry
1214 20:52:19.108668 (+ 213us) spinlock_profiling.cc:243] Waited 7.31 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 000000000098492a 00007ff413306922 00007ff4146e9875 00007ff4146e8399 00007ff4146f07c0 00007ff4146f05cd 0000000000a8970c 00007ff40748e20f 00007ff40749152b 00007ff4074913a8 0000000000a8970c
1214 20:52:19.221497 (+112829us) op_driver.cc:464] REPLICATION: finished
1214 20:52:19.221849 (+ 352us) write_op.cc:301] APPLY: starting
1214 20:52:19.221873 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:19.221936 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:19.221939 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:19.222028 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:19.222071 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:19.222078 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:19.222195 (+ 117us) log.cc:844] Serialized 29 byte log entry
1214 20:52:19.222283 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.222308 (+ 25us) write_op.cc:454] Released schema lock
1214 20:52:19.222316 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:19.222398 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.222401 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":1213824,"child_traces":[["op",{"apply.queue_time_us":292,"apply.run_cpu_time_us":542,"apply.run_wall_time_us":541,"num_ops":2,"prepare.queue_time_us":2051,"prepare.run_cpu_time_us":1110,"prepare.run_wall_time_us":8354,"raft.queue_time_us":1417,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":5,"replication_time_us":120512,"spinlock_wait_cycles":16819968}]]}
I20251214 20:52:19.912999 450 rpcz_store.cc:276] 1214 20:52:18.856742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.856890 (+ 148us) service_pool.cc:225] Handling call
1214 20:52:18.857309 (+ 419us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:18.857527 (+ 218us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:18.857713 (+ 186us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:18.857805 (+ 92us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:19.097248 (+239443us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:19.912852 (+815604us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.881819 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.881976 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.881984 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:18.881986 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:18.882132 (+ 146us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.882147 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.882164 (+ 17us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:18.882249 (+ 85us) tablet.cc:699] Row locks acquired
1214 20:52:18.882251 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:18.882300 (+ 49us) write_op.cc:270] Start()
1214 20:52:18.882360 (+ 60us) write_op.cc:276] Timestamp: P: 1765745538882214 usec, L: 0
1214 20:52:18.882364 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.882625 (+ 261us) log.cc:844] Serialized 405 byte log entry
1214 20:52:19.095655 (+213030us) op_driver.cc:464] REPLICATION: finished
1214 20:52:19.096431 (+ 776us) write_op.cc:301] APPLY: starting
1214 20:52:19.096443 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:19.096488 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:19.096516 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:19.096554 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:19.096581 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:19.096587 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:19.096645 (+ 58us) log.cc:844] Serialized 23 byte log entry
1214 20:52:19.096688 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.096702 (+ 14us) write_op.cc:454] Released schema lock
1214 20:52:19.096708 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:19.096775 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.096779 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":733,"apply.run_cpu_time_us":336,"apply.run_wall_time_us":335,"num_ops":1,"prepare.queue_time_us":23449,"prepare.run_cpu_time_us":902,"prepare.run_wall_time_us":974,"raft.queue_time_us":37,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":213272}]]}
I20251214 20:52:19.914940 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9fa9404adcd247179c0695d6a85e87ea" } modify_external_catalogs: true
I20251214 20:52:19.915115 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9fa9404adcd247179c0695d6a85e87ea" } modify_external_catalogs: true
I20251214 20:52:19.913421 991 leader_election.cc:304] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:19.913292 448 rpcz_store.cc:276] 1214 20:52:18.792597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.792664 (+ 67us) service_pool.cc:225] Handling call
1214 20:52:18.793188 (+ 524us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:18.793321 (+ 133us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:18.793391 (+ 70us) catalog_manager.cc:2824] Locking tablets
1214 20:52:19.098583 (+305192us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:19.227312 (+128729us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:19.227428 (+ 116us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:19.912515 (+685087us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:19.913153 (+ 638us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:19.108767 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:19.108866 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:19.108871 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:19.108873 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:19.109016 (+ 143us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:19.109028 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:19.109033 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:19.109139 (+ 106us) tablet.cc:699] Row locks acquired
1214 20:52:19.109142 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:19.109179 (+ 37us) write_op.cc:270] Start()
1214 20:52:19.109210 (+ 31us) write_op.cc:276] Timestamp: P: 1765745539109147 usec, L: 0
1214 20:52:19.109214 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:19.109360 (+ 146us) log.cc:844] Serialized 756 byte log entry
1214 20:52:19.221582 (+112222us) op_driver.cc:464] REPLICATION: finished
1214 20:52:19.222475 (+ 893us) write_op.cc:301] APPLY: starting
1214 20:52:19.222485 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:19.222513 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:19.222542 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:19.222594 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:19.222618 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:19.222624 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:19.222672 (+ 48us) log.cc:844] Serialized 29 byte log entry
1214 20:52:19.222727 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.222739 (+ 12us) write_op.cc:454] Released schema lock
1214 20:52:19.222743 (+ 4us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:19.222800 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.222803 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":863,"apply.run_cpu_time_us":318,"apply.run_wall_time_us":318,"num_ops":2,"prepare.queue_time_us":9458,"prepare.run_cpu_time_us":682,"prepare.run_wall_time_us":681,"raft.queue_time_us":737,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":112349}]]}
I20251214 20:52:19.921804 448 master_service.cc:954] ReplaceTablet: received request to replace tablet b1f4ea72c6274569a98cab6afdc55d55 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:19.922770 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.923019 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dd657f3baff40bb8cb7f4648367f7a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.926944 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80098b7c24824d129b8f25daf2115460. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.927240 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.928608 1962 raft_consensus.cc:493] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.928647 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.931375 1762 raft_consensus.cc:493] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.931697 1762 raft_consensus.cc:515] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.932307 1762 leader_election.cc:290] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:19.934100 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02e256d4537249a394b80dc36beefef4 consensus state has no leader
I20251214 20:52:19.937194 1977 tablet_replica.cc:333] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:19.937602 1977 raft_consensus.cc:2243] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:19.937992 1977 raft_consensus.cc:2272] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:19.938117 1203 tablet_bootstrap.cc:492] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:19.939281 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a8662a634a4befbd2d6251000d2059" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:19.940086 989 leader_election.cc:343] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.940464 1928 raft_consensus.cc:2804] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:19.941179 1977 ts_tablet_manager.cc:1916] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:19.942659 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7c6fe3b24daca70711a1990af556 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.942963 889 tablet_service.cc:1552] Processing DeleteTablet for tablet 324379e3217442c3addcfe58060b9ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.943212 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd6856b800247de896f16eee66f2827 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c3962c33a294fa98f8bd485268aee71) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:19.943264 454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:50727 (request call id 181) took 1001 ms (client timeout 1000 ms). Trace:
I20251214 20:52:19.943473 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff58f321124d12b5330dca185d426e with delete_type TABLET_DATA_DELETED (replaced by tablet 42bdcecba5534294aca1eb419f18e9d2) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.943676 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:19.943375 454 rpcz_store.cc:269] 1214 20:52:18.941600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.108811 (+167211us) service_pool.cc:225] Handling call
1214 20:52:19.943203 (+834392us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:19.120815 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:19.120921 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:19.120927 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:19.120929 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:19.121040 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:19.121052 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:19.121056 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:19.121155 (+ 99us) tablet.cc:699] Row locks acquired
1214 20:52:19.121157 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:19.121199 (+ 42us) write_op.cc:270] Start()
1214 20:52:19.121233 (+ 34us) write_op.cc:276] Timestamp: P: 1765745539121164 usec, L: 0
1214 20:52:19.121237 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:19.121445 (+ 208us) log.cc:844] Serialized 901 byte log entry
1214 20:52:19.221680 (+100235us) op_driver.cc:464] REPLICATION: finished
1214 20:52:19.223389 (+ 1709us) write_op.cc:301] APPLY: starting
1214 20:52:19.223399 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:19.223445 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:19.223466 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:19.223513 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:19.223538 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:19.223545 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:19.223593 (+ 48us) log.cc:844] Serialized 29 byte log entry
1214 20:52:19.223645 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.223656 (+ 11us) write_op.cc:454] Released schema lock
1214 20:52:19.223661 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:19.223721 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.223724 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":23,"spinlock_wait_cycles":34048,"child_traces":[["op",{"apply.queue_time_us":1679,"apply.run_cpu_time_us":326,"apply.run_wall_time_us":325,"num_ops":2,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":734,"prepare.run_wall_time_us":735,"raft.queue_time_us":14508,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":100426}]]}
I20251214 20:52:19.943898 880 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.944083 454 master_service.cc:954] ReplaceTablet: received request to replace tablet 2b48bb260e854beb9ec61ce9c79b46c2 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:19.944120 894 tablet_service.cc:1552] Processing DeleteTablet for tablet b652a8a80246443491551e76e68d2a8b with delete_type TABLET_DATA_DELETED (replaced by tablet 18e119bc44874aecb4e83c8d6040c8cd) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.944289 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d74d3abc45a41969289c0a0892bfa04 with delete_type TABLET_DATA_DELETED (replaced by tablet 727e7b61ae3942098712720d344c5310) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.944491 882 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a2143b5508445492b970d2588cf1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.945849 883 tablet_service.cc:1552] Processing DeleteTablet for tablet a0279992d1f2419ab1caae116a0391b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.945897 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 9523d72fd91445539d67e1eac2a80afd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.946437 881 tablet_service.cc:1552] Processing DeleteTablet for tablet c16895ca08d94b56b5c4bc9352e330ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.946537 1928 raft_consensus.cc:493] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:19.946615 879 tablet_service.cc:1552] Processing DeleteTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.946717 895 tablet_service.cc:1552] Processing DeleteTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf with delete_type TABLET_DATA_DELETED (replaced by tablet 6fef1d6add764fd28a8f4995b337ea71) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.946784 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dd657f3baff40bb8cb7f4648367f7a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:19.946822 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 40a9a6192c744282ab5672cbe15fd299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.951447 1203 ts_tablet_manager.cc:1403] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.318s user 0.007s sys 0.008s
I20251214 20:52:19.952128 1206 tablet_bootstrap.cc:654] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:19.953132 750 tablet_service.cc:1552] Processing DeleteTablet for tablet fe4086fc7a4f4db0b7541f06c9833aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:19.953826 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d72f55d2d2440d993aa205d96c356f1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:19.954638 1928 raft_consensus.cc:3060] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:19.955678 1977 ts_tablet_manager.cc:1929] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:19.955905 1977 log.cc:1199] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/fe4086fc7a4f4db0b7541f06c9833aab
I20251214 20:52:19.957041 1977 ts_tablet_manager.cc:1950] T fe4086fc7a4f4db0b7541f06c9833aab P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:19.958724 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet fe4086fc7a4f4db0b7541f06c9833aab (table default.Table_8076bc8b31ba411faf089f314e6ed2ef [id=25016f04bb864f92a1cdfb6ceaa9fba9]) successfully deleted
W20251214 20:52:19.959615 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 98b7cac3930a4a9eb16ac413c8f1e092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98b7cac3930a4a9eb16ac413c8f1e092 already in progress: creating tablet
W20251214 20:52:19.960505 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 7dd657f3baff40bb8cb7f4648367f7a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dd657f3baff40bb8cb7f4648367f7a1 already in progress: creating tablet
I20251214 20:52:19.961020 1977 tablet_replica.cc:333] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:19.961582 1762 raft_consensus.cc:493] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.961794 1762 raft_consensus.cc:515] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.962563 1762 leader_election.cc:290] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:19.963835 1977 raft_consensus.cc:2243] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:19.964095 1977 raft_consensus.cc:2272] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:19.964862 1977 ts_tablet_manager.cc:1916] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:19.967733 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 292a28b12df246a8bc855d81c530746e
I20251214 20:52:19.968605 742 tablet_service.cc:1505] Processing CreateTablet for tablet 80098b7c24824d129b8f25daf2115460 (DEFAULT_TABLE table=default.Table_8ca3b0a5a22a425b866b0b2565f80fac [id=2a055be9028342be97f5c98f03f5a137]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.969038 450 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 96) took 1031 ms. Trace:
I20251214 20:52:19.969528 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 292a28b12df246a8bc855d81c530746e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:19.969149 450 rpcz_store.cc:276] 1214 20:52:18.937725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.914750 (+977025us) service_pool.cc:225] Handling call
1214 20:52:19.915224 (+ 474us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:19.915376 (+ 152us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:19.915446 (+ 70us) catalog_manager.cc:2824] Locking tablets
1214 20:52:19.915599 (+ 153us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:19.927033 (+ 11434us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:19.967606 (+ 40573us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:19.967650 (+ 44us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:19.969002 (+ 1352us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:19.919605 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:19.919753 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:19.919760 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:19.919763 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:19.919899 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:19.919917 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:19.919923 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:19.920037 (+ 114us) tablet.cc:699] Row locks acquired
1214 20:52:19.920039 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:19.920086 (+ 47us) write_op.cc:270] Start()
1214 20:52:19.920171 (+ 85us) write_op.cc:276] Timestamp: P: 1765745539919991 usec, L: 0
1214 20:52:19.920175 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:19.920408 (+ 233us) log.cc:844] Serialized 725 byte log entry
1214 20:52:19.924396 (+ 3988us) op_driver.cc:464] REPLICATION: finished
1214 20:52:19.924851 (+ 455us) write_op.cc:301] APPLY: starting
1214 20:52:19.924868 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:19.924914 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:19.924938 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:19.925007 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:19.925045 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:19.925052 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:19.925131 (+ 79us) log.cc:844] Serialized 29 byte log entry
1214 20:52:19.925193 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.925216 (+ 23us) write_op.cc:454] Released schema lock
1214 20:52:19.925223 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:19.925298 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.925301 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":389,"apply.run_cpu_time_us":444,"apply.run_wall_time_us":444,"num_ops":2,"prepare.queue_time_us":2986,"prepare.run_cpu_time_us":942,"prepare.run_wall_time_us":943,"raft.queue_time_us":433,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":262,"replication_time_us":4191}]]}
I20251214 20:52:19.970286 739 tablet_service.cc:1505] Processing CreateTablet for tablet 6eca34627bbc4a3f9915be72f3128370 (DEFAULT_TABLE table=default.Table_a91064ac3fb94282a02fbc867fdd581c [id=6b97974ca1764265aa6c291f8439da8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.971115 755 tablet_service.cc:1505] Processing CreateTablet for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (DEFAULT_TABLE table=default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.971393 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dd657f3baff40bb8cb7f4648367f7a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:19.971551 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:19.971684 752 tablet_service.cc:1505] Processing CreateTablet for tablet 727e7b61ae3942098712720d344c5310 (DEFAULT_TABLE table=default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.972235 455 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 85) took 1117 ms. Trace:
I20251214 20:52:19.972239 754 tablet_service.cc:1505] Processing CreateTablet for tablet dfe50a6219134431b0a025317932609b (DEFAULT_TABLE table=default.Table_b5f628ed069a4b19996c9b579d7678bc [id=a03c04d94997449bac7447ed634577d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.972558 756 tablet_service.cc:1505] Processing CreateTablet for tablet a07d94d5a431488489ee9fbcc113a9d7 (DEFAULT_TABLE table=default.Table_493f60b9d8024509856ed934d379f829 [id=a40a329d98674364aa3794d3a07fdeb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.972337 455 rpcz_store.cc:276] 1214 20:52:18.854705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.854763 (+ 58us) service_pool.cc:225] Handling call
1214 20:52:18.855337 (+ 574us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:18.855587 (+ 250us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:18.855823 (+ 236us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:18.855887 (+ 64us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:19.096417 (+240530us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:19.972185 (+875768us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.873860 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.873974 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.873980 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:18.873982 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:18.874078 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.874089 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.874093 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:18.874204 (+ 111us) tablet.cc:699] Row locks acquired
1214 20:52:18.874207 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:18.874242 (+ 35us) write_op.cc:270] Start()
1214 20:52:18.874273 (+ 31us) write_op.cc:276] Timestamp: P: 1765745538874162 usec, L: 0
1214 20:52:18.874276 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.874432 (+ 156us) log.cc:844] Serialized 405 byte log entry
1214 20:52:19.095594 (+221162us) op_driver.cc:464] REPLICATION: finished
1214 20:52:19.095936 (+ 342us) write_op.cc:301] APPLY: starting
1214 20:52:19.095948 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:19.095972 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:19.095992 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:19.096046 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:19.096065 (+ 19us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:19.096071 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:19.096137 (+ 66us) log.cc:844] Serialized 23 byte log entry
1214 20:52:19.096200 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.096221 (+ 21us) write_op.cc:454] Released schema lock
1214 20:52:19.096230 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:19.096312 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.096317 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":298,"apply.run_cpu_time_us":367,"apply.run_wall_time_us":367,"mutex_wait_us":1316,"num_ops":1,"prepare.queue_time_us":17399,"prepare.run_cpu_time_us":695,"prepare.run_wall_time_us":7919,"raft.queue_time_us":7190,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":221296}]]}
I20251214 20:52:19.973097 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ba7f3997d83f41b3a3a7019d61267c0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:19.973557 1964 raft_consensus.cc:493] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.973619 1977 ts_tablet_manager.cc:1929] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251214 20:52:19.973665 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba7f3997d83f41b3a3a7019d61267c0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:19.973774 1977 log.cc:1199] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/604b7c6fe3b24daca70711a1990af556
I20251214 20:52:19.974134 1977 ts_tablet_manager.cc:1950] T 604b7c6fe3b24daca70711a1990af556 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:19.974867 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80098b7c24824d129b8f25daf2115460. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.975279 451 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 87) took 1115 ms. Trace:
I20251214 20:52:19.975504 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 604b7c6fe3b24daca70711a1990af556 (table default.Table_fbb5b21626c142c8baac60a7f29d3c51 [id=7cb78d704474480db14d212ee89166a0]) successfully deleted
I20251214 20:52:19.975982 1203 raft_consensus.cc:359] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.975374 451 rpcz_store.cc:276] 1214 20:52:18.859809 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.859874 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:18.860250 (+ 376us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:18.860444 (+ 194us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:18.860671 (+ 227us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:18.860749 (+ 78us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:19.100266 (+239517us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:19.975210 (+874944us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:18.882812 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:18.882902 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:18.882907 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:18.882909 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:18.882988 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:18.882997 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:18.883001 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:18.883104 (+ 103us) tablet.cc:699] Row locks acquired
1214 20:52:18.883106 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:18.883171 (+ 65us) write_op.cc:270] Start()
1214 20:52:18.883200 (+ 29us) write_op.cc:276] Timestamp: P: 1765745538883088 usec, L: 0
1214 20:52:18.883203 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:18.883342 (+ 139us) log.cc:844] Serialized 405 byte log entry
1214 20:52:19.095721 (+212379us) op_driver.cc:464] REPLICATION: finished
1214 20:52:19.096842 (+ 1121us) write_op.cc:301] APPLY: starting
1214 20:52:19.096854 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:19.096881 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:19.096910 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:19.096941 (+ 31us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:19.096965 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:19.096972 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:19.097048 (+ 76us) log.cc:844] Serialized 23 byte log entry
1214 20:52:19.097088 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.097100 (+ 12us) write_op.cc:454] Released schema lock
1214 20:52:19.097106 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1083,"apply.run_cpu_time_us":318,"apply.run_wall_time_us":318,"num_ops":1,"prepare.queue_time_us":21532,"prepare.run_cpu_time_us":657,"prepare.run_wall_time_us":715,"raft.queue_time_us":35,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":212499,"spinlock_wait_cycles":82688,"thread_start_us":11487,"threads_started":1}]]}
I20251214 20:52:19.976290 1203 raft_consensus.cc:385] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:19.976385 1203 raft_consensus.cc:740] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:19.976613 1203 consensus_queue.cc:260] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.976564 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_2718b41c3d5643439f3a0d867fbb8116"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:19.977228 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2718b41c3d5643439f3a0d867fbb8116 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:19.978192 1203 ts_tablet_manager.cc:1434] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20251214 20:52:19.979315 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3962c33a294fa98f8bd485268aee71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:19.979521 1964 raft_consensus.cc:515] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.980178 1964 leader_election.cc:290] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:19.980351 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:19.980516 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb698ff30fcc47d09000cc68d2c5225b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:19.980811 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf04f91e79754569a51d468ab576b0dc" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:19.980911 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a07d94d5a431488489ee9fbcc113a9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.983036 1977 tablet_replica.cc:333] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:19.983296 1977 raft_consensus.cc:2243] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:19.983510 1977 raft_consensus.cc:2272] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:19.983544 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:19.984139 1977 ts_tablet_manager.cc:1916] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:19.987908 848 leader_election.cc:343] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.988684 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d72f55d2d2440d993aa205d96c356f1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:19.988888 451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 100) took 1021 ms. Trace:
I20251214 20:52:19.988981 451 rpcz_store.cc:276] 1214 20:52:18.967825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.976290 (+1008465us) service_pool.cc:225] Handling call
1214 20:52:19.977368 (+ 1078us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:19.978004 (+ 636us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:19.988616 (+ 10612us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:19.988736 (+ 120us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:19.988828 (+ 92us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:19.984004 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:19.984117 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:19.984123 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:19.984125 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:19.984207 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:19.984220 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:19.984225 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:19.984315 (+ 90us) tablet.cc:699] Row locks acquired
1214 20:52:19.984317 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:19.984350 (+ 33us) write_op.cc:270] Start()
1214 20:52:19.984378 (+ 28us) write_op.cc:276] Timestamp: P: 1765745539984247 usec, L: 0
1214 20:52:19.984381 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:19.984542 (+ 161us) log.cc:844] Serialized 433 byte log entry
1214 20:52:19.987533 (+ 2991us) op_driver.cc:464] REPLICATION: finished
1214 20:52:19.987656 (+ 123us) write_op.cc:301] APPLY: starting
1214 20:52:19.987668 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:19.987714 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:19.987748 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:19.987852 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:19.987884 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:19.987892 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:19.987955 (+ 63us) log.cc:844] Serialized 29 byte log entry
1214 20:52:19.988008 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.988024 (+ 16us) write_op.cc:454] Released schema lock
1214 20:52:19.988030 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:19.988095 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.988098 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":431,"num_ops":2,"prepare.queue_time_us":5451,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":629,"raft.queue_time_us":448,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":214,"replication_time_us":3126}]]}
I20251214 20:52:19.990108 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_26a82e34bb024bd89d6819c80056ba3b" } new_table_name: "default.Table_13f016135de44cf78187fbc4f9f8c622" modify_external_catalogs: true
W20251214 20:52:19.992260 848 leader_election.cc:343] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.993145 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfe50a6219134431b0a025317932609b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:19.993237 851 leader_election.cc:343] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:19.993399 851 leader_election.cc:304] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:19.993835 1762 raft_consensus.cc:2749] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:19.994127 1856 ts_tablet_manager.cc:1929] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:19.996764 1977 ts_tablet_manager.cc:1929] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:19.996893 1977 log.cc:1199] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/324379e3217442c3addcfe58060b9ad7
I20251214 20:52:19.997319 1977 ts_tablet_manager.cc:1950] T 324379e3217442c3addcfe58060b9ad7 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:19.998656 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 324379e3217442c3addcfe58060b9ad7 (table default.Table_ff05c589ffd8444daa99c956e92c37e6 [id=16a8869e1ba641438d0836f6f204414f]) successfully deleted
W20251214 20:52:19.999210 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d01db63fd5d2474a9defdce0b98fc29d consensus state has no leader
I20251214 20:52:19.999203 1962 raft_consensus.cc:515] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:19.999436 451 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 101) took 1028 ms. Trace:
I20251214 20:52:19.999696 1963 raft_consensus.cc:493] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:19.973696 747 tablet_service.cc:1505] Processing CreateTablet for tablet c8e33ba1149a44c294885d76f676160a (DEFAULT_TABLE table=default.Table_3f7396ac07d649fa8d342ae05765da80 [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:19.999722 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "88b9ca42a5ce4572aebe2a24f4758b12" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:19.999518 451 rpcz_store.cc:276] 1214 20:52:18.970883 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.989923 (+1019040us) service_pool.cc:225] Handling call
1214 20:52:19.990278 (+ 355us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:19.990479 (+ 201us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:19.990646 (+ 167us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:19.990744 (+ 98us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:19.998800 (+ 8056us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:19.999380 (+ 580us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:19.991296 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:19.991381 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:19.991386 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:19.991388 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:19.991474 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:19.991485 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:19.991490 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:19.991541 (+ 51us) tablet.cc:699] Row locks acquired
1214 20:52:19.991543 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:19.991573 (+ 30us) write_op.cc:270] Start()
1214 20:52:19.991599 (+ 26us) write_op.cc:276] Timestamp: P: 1765745539991469 usec, L: 0
1214 20:52:19.991602 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:19.991774 (+ 172us) log.cc:844] Serialized 405 byte log entry
1214 20:52:19.996507 (+ 4733us) op_driver.cc:464] REPLICATION: finished
1214 20:52:19.996635 (+ 128us) write_op.cc:301] APPLY: starting
1214 20:52:19.996649 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:19.996716 (+ 67us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:19.996743 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:19.996791 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:19.996824 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:19.996832 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:19.996901 (+ 69us) log.cc:844] Serialized 23 byte log entry
1214 20:52:19.996952 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.996970 (+ 18us) write_op.cc:454] Released schema lock
1214 20:52:19.996976 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:19.997051 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.997055 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":84,"apply.run_cpu_time_us":398,"apply.run_wall_time_us":397,"num_ops":1,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":557,"prepare.run_wall_time_us":556,"raft.queue_time_us":55,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":4873}]]}
I20251214 20:52:20.000809 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_30e486c0b499465c89d73644d7980f81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:20.001345 1928 raft_consensus.cc:515] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:20.001642 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30e486c0b499465c89d73644d7980f81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.002231 1856 log.cc:1199] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/f3a2143b5508445492b970d2588cf1cf
I20251214 20:52:20.002262 1203 tablet_bootstrap.cc:492] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:20.002398 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e33ba1149a44c294885d76f676160a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.003034 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 99) took 1035 ms. Trace:
I20251214 20:52:20.003309 1856 ts_tablet_manager.cc:1950] T f3a2143b5508445492b970d2588cf1cf P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.003247 455 rpcz_store.cc:276] 1214 20:52:18.967665 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.972893 (+1005228us) service_pool.cc:225] Handling call
1214 20:52:19.973763 (+ 870us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:19.974211 (+ 448us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:20.002809 (+ 28598us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:20.002901 (+ 92us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:20.002980 (+ 79us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:19.976053 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:19.976193 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:19.976200 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:19.976202 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:19.976294 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:19.976308 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:19.976313 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:19.976390 (+ 77us) tablet.cc:699] Row locks acquired
1214 20:52:19.976392 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:19.976430 (+ 38us) write_op.cc:270] Start()
1214 20:52:19.976466 (+ 36us) write_op.cc:276] Timestamp: P: 1765745539976327 usec, L: 0
1214 20:52:19.976470 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:19.976705 (+ 235us) log.cc:844] Serialized 433 byte log entry
1214 20:52:19.980160 (+ 3455us) op_driver.cc:464] REPLICATION: finished
1214 20:52:19.981402 (+ 1242us) write_op.cc:301] APPLY: starting
1214 20:52:19.981419 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:19.981468 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:19.981504 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:19.981637 (+ 133us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:19.981672 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:19.981690 (+ 18us) write_op.cc:312] APPLY: finished
1214 20:52:19.981772 (+ 82us) log.cc:844] Serialized 29 byte log entry
1214 20:52:19.981831 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.981852 (+ 21us) write_op.cc:454] Released schema lock
1214 20:52:19.981858 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:19.981925 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:19.981929 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1178,"apply.run_cpu_time_us":522,"apply.run_wall_time_us":522,"num_ops":2,"prepare.queue_time_us":1368,"prepare.run_cpu_time_us":779,"prepare.run_wall_time_us":779,"raft.queue_time_us":408,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":256,"replication_time_us":3661,"spinlock_wait_cycles":18816}]]}
I20251214 20:52:20.004051 745 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b7149a436a4cab80b39adccf8567f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.004573 1963 raft_consensus.cc:515] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.004925 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 727e7b61ae3942098712720d344c5310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.004854 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_dc35e103390944548c36be9ba4900882"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.005399 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc35e103390944548c36be9ba4900882 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.005618 1962 leader_election.cc:290] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.006253 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6eca34627bbc4a3f9915be72f3128370. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.007200 774 raft_consensus.cc:3060] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:20.008198 712 leader_election.cc:343] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: fe4086fc7a4f4db0b7541f06c9833aab
I20251214 20:52:20.008397 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.008847 751 tablet_service.cc:1552] Processing DeleteTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.010025 1963 leader_election.cc:290] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.010087 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8af99e0bd8774a678575847a9ec2e557" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:20.010412 904 raft_consensus.cc:2393] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:20.013880 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 38) took 1241 ms. Trace:
I20251214 20:52:20.014112 1856 rpcz_store.cc:276] 1214 20:52:18.762569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.762603 (+ 34us) service_pool.cc:225] Handling call
1214 20:52:19.679670 (+917067us) ts_tablet_manager.cc:1151] Deleting tablet f3a2143b5508445492b970d2588cf1cf
1214 20:52:19.679680 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:19.901480 (+221800us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:19.993969 (+ 92489us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.004143 (+ 10174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":916819}
I20251214 20:52:20.014767 1856 tablet_replica.cc:333] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.015116 1856 raft_consensus.cc:2243] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.016263 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a8662a634a4befbd2d6251000d2059" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:20.016481 904 raft_consensus.cc:3060] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.017894 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af99e0bd8774a678575847a9ec2e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.019606 1856 raft_consensus.cc:2272] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.019865 1928 leader_election.cc:290] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.020519 1856 ts_tablet_manager.cc:1916] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.021358 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 9523d72fd91445539d67e1eac2a80afd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.021416 455 master_service.cc:954] ReplaceTablet: received request to replace tablet 25974e5db555447baf5799775d15cd14 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:20.021638 904 raft_consensus.cc:2468] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:20.021870 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a7fc15882e2474bb6885070be13b1fa with delete_type TABLET_DATA_DELETED (replaced by tablet 65e79db74fa046698507c83416ee1669) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.022069 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.022267 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff with delete_type TABLET_DATA_DELETED (replaced by tablet 6a000951f61948ac85fdd431f3d162c2) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.022239 991 leader_election.cc:304] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:20.022442 451 master_service.cc:954] ReplaceTablet: received request to replace tablet 5da9deeedce0421c9aeaebd71ba11b1b from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:20.022455 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 52daf1204bf142e7b82852bdec6582fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.024227 746 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.027050 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.027944 746 tablet_service.cc:1552] Processing DeleteTablet for tablet b652a8a80246443491551e76e68d2a8b with delete_type TABLET_DATA_DELETED (replaced by tablet 18e119bc44874aecb4e83c8d6040c8cd) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.028609 746 tablet_service.cc:1552] Processing DeleteTablet for tablet b315744c8cd043c5a8bdceec73a7e28b with delete_type TABLET_DATA_DELETED (replaced by tablet f945814952b14131ba4e0c67a403a680) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.028597 1203 tablet_bootstrap.cc:654] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.029438 751 tablet_service.cc:1552] Processing DeleteTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.029633 745 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a2143b5508445492b970d2588cf1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.030817 753 tablet_service.cc:1552] Processing DeleteTablet for tablet f0d661a1bb034053a4634ff537eafc7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.032009 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb698ff30fcc47d09000cc68d2c5225b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:20.032544 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet f3a2143b5508445492b970d2588cf1cf (table default.Table_d653f9c533784976ac955f90dc39eb56 [id=295a34df92b44b738e3aa3d1d8195673]) successfully deleted
W20251214 20:52:20.032691 851 leader_election.cc:343] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.032853 851 leader_election.cc:304] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:20.032915 753 tablet_service.cc:1552] Processing DeleteTablet for tablet bb698ff30fcc47d09000cc68d2c5225b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.033325 1762 raft_consensus.cc:2749] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.033622 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 292a28b12df246a8bc855d81c530746e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.033947 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 324379e3217442c3addcfe58060b9ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.034247 1928 raft_consensus.cc:2804] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:20.034512 1928 raft_consensus.cc:697] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:20.034832 1928 consensus_queue.cc:237] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.035391 1977 tablet_replica.cc:333] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.035640 1977 raft_consensus.cc:2243] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.035985 1977 raft_consensus.cc:2272] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.037911 1977 ts_tablet_manager.cc:1916] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.041677 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd6856b800247de896f16eee66f2827 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c3962c33a294fa98f8bd485268aee71) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.042481 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 2b48bb260e854beb9ec61ce9c79b46c2
I20251214 20:52:20.043157 454 catalog_manager.cc:6724] ReplaceTablet: tablet 2b48bb260e854beb9ec61ce9c79b46c2 deleted and replaced by tablet 566c6e4761d5486d964e276d151e32b0
I20251214 20:52:20.043311 454 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 98) took 1092 ms. Trace:
I20251214 20:52:20.043726 753 tablet_service.cc:1552] Processing DeleteTablet for tablet bd8226b431c6456885e928880938969d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.043411 454 rpcz_store.cc:276] 1214 20:52:18.950626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.944017 (+993391us) service_pool.cc:225] Handling call
1214 20:52:20.043263 (+ 99246us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:20.036020 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.036135 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.036140 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:20.036142 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:20.036235 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.036247 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.036251 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:20.036340 (+ 89us) tablet.cc:699] Row locks acquired
1214 20:52:20.036342 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:20.036376 (+ 34us) write_op.cc:270] Start()
1214 20:52:20.036402 (+ 26us) write_op.cc:276] Timestamp: P: 1765745540036261 usec, L: 0
1214 20:52:20.036406 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.036556 (+ 150us) log.cc:844] Serialized 656 byte log entry
1214 20:52:20.041241 (+ 4685us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.041376 (+ 135us) write_op.cc:301] APPLY: starting
1214 20:52:20.041393 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.041438 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.041474 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.041580 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.041646 (+ 66us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:20.041653 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:20.041727 (+ 74us) log.cc:844] Serialized 29 byte log entry
1214 20:52:20.041787 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.041805 (+ 18us) write_op.cc:454] Released schema lock
1214 20:52:20.041811 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:20.041881 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.041885 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":768,"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":498,"apply.run_wall_time_us":497,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":653,"prepare.run_wall_time_us":1232,"raft.queue_time_us":55,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":210,"replication_time_us":4803}]]}
I20251214 20:52:20.043943 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b48bb260e854beb9ec61ce9c79b46c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 566c6e4761d5486d964e276d151e32b0) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.044700 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_8163ed145f64447c8eb6fc0f8d0a1d4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.045526 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8163ed145f64447c8eb6fc0f8d0a1d4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.046075 753 tablet_service.cc:1552] Processing DeleteTablet for tablet c25832da5c1644329e18653000f3330e with delete_type TABLET_DATA_DELETED (replaced by tablet 0c26d1f969ed4071a2f2cf1615a17cac) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.047012 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b48bb260e854beb9ec61ce9c79b46c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 566c6e4761d5486d964e276d151e32b0) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.048336 1762 raft_consensus.cc:493] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.048519 1762 raft_consensus.cc:515] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:20.049002 1856 ts_tablet_manager.cc:1929] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.049132 1856 log.cc:1199] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/472e4a5990e64d02b1d52dc6e86d7190
I20251214 20:52:20.049268 1762 leader_election.cc:290] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:20.049513 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.049551 1856 ts_tablet_manager.cc:1950] T 472e4a5990e64d02b1d52dc6e86d7190 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.049930 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcdd6bafa5c4aa6b59668cde523d010" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.050195 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcdd6bafa5c4aa6b59668cde523d010" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:20.050863 848 leader_election.cc:343] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.051810 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 242a77ae57174de48f4eaf43384ee0ba
I20251214 20:52:20.052268 1203 tablet_bootstrap.cc:492] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:20.052330 453 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 95) took 1118 ms. Trace:
I20251214 20:52:20.052416 1203 ts_tablet_manager.cc:1403] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.009s
I20251214 20:52:20.052461 453 rpcz_store.cc:276] 1214 20:52:18.933940 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.878026 (+944086us) service_pool.cc:225] Handling call
1214 20:52:19.878636 (+ 610us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:19.878779 (+ 143us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:19.878843 (+ 64us) catalog_manager.cc:2824] Locking tablets
1214 20:52:20.035347 (+156504us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:20.051618 (+ 16271us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:20.051695 (+ 77us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:20.051738 (+ 43us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:20.052291 (+ 553us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:20.037276 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.037363 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.037367 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:20.037369 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:20.037888 (+ 519us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.037897 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.037902 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:20.037986 (+ 84us) tablet.cc:699] Row locks acquired
1214 20:52:20.037989 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:20.038018 (+ 29us) write_op.cc:270] Start()
1214 20:52:20.038042 (+ 24us) write_op.cc:276] Timestamp: P: 1765745540037905 usec, L: 0
1214 20:52:20.038046 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.038179 (+ 133us) log.cc:844] Serialized 725 byte log entry
1214 20:52:20.050966 (+ 12787us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.051150 (+ 184us) write_op.cc:301] APPLY: starting
1214 20:52:20.051164 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.051209 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.051249 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.051336 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.051379 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:20.051386 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:20.051457 (+ 71us) log.cc:844] Serialized 29 byte log entry
1214 20:52:20.051514 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.051534 (+ 20us) write_op.cc:454] Released schema lock
1214 20:52:20.051540 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:20.051626 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.051630 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":12032,"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":484,"apply.run_wall_time_us":483,"mutex_wait_us":82,"num_ops":2,"prepare.queue_time_us":1309,"prepare.run_cpu_time_us":590,"prepare.run_wall_time_us":1281,"raft.queue_time_us":31,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":12890}]]}
I20251214 20:52:20.053241 1203 raft_consensus.cc:359] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.053274 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5242f3b282284ab7aff8525106732d53" } new_table_name: "default.Table_67fa945ac69a4a599f3dd43cfa93c0ff" modify_external_catalogs: true
I20251214 20:52:20.053424 1203 raft_consensus.cc:385] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.053509 1203 raft_consensus.cc:740] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:20.055409 851 leader_election.cc:343] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.055564 851 leader_election.cc:304] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:20.055759 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 242a77ae57174de48f4eaf43384ee0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.055968 1762 raft_consensus.cc:2749] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.056767 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 242a77ae57174de48f4eaf43384ee0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.059823 1203 consensus_queue.cc:260] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.060029 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b4f582d8165439d86d9228c146dc41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.060489 1203 ts_tablet_manager.cc:1434] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20251214 20:52:20.060822 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 43) took 1297 ms. Trace:
I20251214 20:52:20.061146 1203 tablet_bootstrap.cc:492] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:20.061753 753 tablet_service.cc:1552] Processing DeleteTablet for tablet a063083f54ad49d2a842fa080d44ed85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.061430 1856 rpcz_store.cc:276] 1214 20:52:18.763203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.763313 (+ 110us) service_pool.cc:225] Handling call
1214 20:52:20.014652 (+1251339us) ts_tablet_manager.cc:1151] Deleting tablet 472e4a5990e64d02b1d52dc6e86d7190
1214 20:52:20.014662 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.038322 (+ 23660us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.048836 (+ 10514us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.060778 (+ 11942us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251155}
I20251214 20:52:20.062713 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 33) took 1309 ms. Trace:
I20251214 20:52:20.062928 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 472e4a5990e64d02b1d52dc6e86d7190 (table default.Table_2acb51a56a60469e941100c99494a99b [id=b16533ff3115482d9357bd88bc1406cd]) successfully deleted
I20251214 20:52:20.063617 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.064329 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 2da66ec83e9140f7a605f0c86abff468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.065043 1977 ts_tablet_manager.cc:1929] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.065217 1977 log.cc:1199] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/6bd6856b800247de896f16eee66f2827
I20251214 20:52:20.065621 1977 ts_tablet_manager.cc:1950] T 6bd6856b800247de896f16eee66f2827 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.064577 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.064851 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 35a529702f924a5c9bf53c3c668c6094 with delete_type TABLET_DATA_DELETED (replaced by tablet 719d157ac8f74eb7911df05ab5e4a3d5) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.065853 1856 rpcz_store.cc:276] 1214 20:52:18.752763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.752851 (+ 88us) service_pool.cc:225] Handling call
1214 20:52:20.062496 (+1309645us) ts_tablet_manager.cc:1151] Deleting tablet 2a7fc15882e2474bb6885070be13b1fa
1214 20:52:20.062505 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.062669 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1298763}
I20251214 20:52:20.065326 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 1dca369dc5f54a4ea1783e334c5670d6) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.065596 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 63497e77baeb4d328be2421b5732a28a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.066422 1856 tablet_replica.cc:333] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:20.066679 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2a7fc15882e2474bb6885070be13b1fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a7fc15882e2474bb6885070be13b1fa already in progress: creating tablet
I20251214 20:52:20.066779 1856 raft_consensus.cc:2243] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.066886 755 tablet_service.cc:1552] Processing DeleteTablet for tablet ca263a80702f4f64aa7c872c2be08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.067168 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 292a28b12df246a8bc855d81c530746e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.067272 1856 raft_consensus.cc:2272] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:20.067689 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88b9ca42a5ce4572aebe2a24f4758b12 consensus state has no leader
I20251214 20:52:20.068151 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d20c0b0d957a47c3985941ceda684666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.069128 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d20c0b0d957a47c3985941ceda684666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.063871 1203 tablet_bootstrap.cc:654] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.070127 1856 ts_tablet_manager.cc:1916] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.070449 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.072054 752 tablet_service.cc:1552] Processing DeleteTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.072640 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 6bd6856b800247de896f16eee66f2827 (table default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]) successfully deleted
I20251214 20:52:20.073181 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf04f91e79754569a51d468ab576b0dc" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:20.073987 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe4086fc7a4f4db0b7541f06c9833aab" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:20.072365 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b48bb260e854beb9ec61ce9c79b46c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 566c6e4761d5486d964e276d151e32b0) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.073920 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 242a77ae57174de48f4eaf43384ee0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.074951 739 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b22043d81b493fb12116422b1d3fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet ba7952290e2c4746802cffeb14971da0) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.075562 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_a218538b99ae4cf6b55add7cf3b995a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.076337 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a218538b99ae4cf6b55add7cf3b995a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.079528 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a8662a634a4befbd2d6251000d2059" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:20.083487 1977 tablet_replica.cc:333] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.083914 1977 raft_consensus.cc:2243] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.084992 448 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 97) took 1142 ms. Trace:
I20251214 20:52:20.085180 448 rpcz_store.cc:276] 1214 20:52:18.942607 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.921724 (+979117us) service_pool.cc:225] Handling call
1214 20:52:20.084946 (+163222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 20:52:20.085469 1977 raft_consensus.cc:2272] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.085602 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_a941092a7b2944bfa37073939660b587"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.085994 713 leader_election.cc:343] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: bd8226b431c6456885e928880938969d
I20251214 20:52:20.086126 713 leader_election.cc:304] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:20.086201 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a941092a7b2944bfa37073939660b587 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.086242 1977 ts_tablet_manager.cc:1916] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.086463 1963 raft_consensus.cc:2749] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.087841 713 leader_election.cc:304] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:20.089174 1963 raft_consensus.cc:3060] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.091933 1963 raft_consensus.cc:2749] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.092890 1856 ts_tablet_manager.cc:1929] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.092944 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_e74a0c96be654f4c808071da1ae734a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.093535 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e74a0c96be654f4c808071da1ae734a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.097442 713 leader_election.cc:304] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:20.097759 1856 log.cc:1199] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/63497e77baeb4d328be2421b5732a28a
I20251214 20:52:20.098441 451 master_service.cc:954] ReplaceTablet: received request to replace tablet 65e79db74fa046698507c83416ee1669 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:20.098634 1856 ts_tablet_manager.cc:1950] T 63497e77baeb4d328be2421b5732a28a P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.099990 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 39) took 1337 ms. Trace:
I20251214 20:52:20.101572 1856 rpcz_store.cc:276] 1214 20:52:18.762664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.762888 (+ 224us) service_pool.cc:225] Handling call
1214 20:52:20.066354 (+1303466us) ts_tablet_manager.cc:1151] Deleting tablet 63497e77baeb4d328be2421b5732a28a
1214 20:52:20.066364 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.088437 (+ 22073us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.092737 (+ 4300us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.099929 (+ 7192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1283733}
I20251214 20:52:20.102208 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 63497e77baeb4d328be2421b5732a28a (table default.Table_d553ca568da242899d768f9848baba3d [id=376e4eabd04d4a53ab2772323b8c41b7]) successfully deleted
I20251214 20:52:20.102798 1977 ts_tablet_manager.cc:1929] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:20.102321 1856 tablet_replica.cc:333] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.103219 1856 raft_consensus.cc:2243] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.103430 1977 log.cc:1199] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/24ff58f321124d12b5330dca185d426e
I20251214 20:52:20.103739 1856 raft_consensus.cc:2272] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.104211 1977 ts_tablet_manager.cc:1950] T 24ff58f321124d12b5330dca185d426e P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.104950 1856 ts_tablet_manager.cc:1916] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.109720 713 leader_election.cc:304] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:20.110101 1962 raft_consensus.cc:2749] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:20.110708 713 leader_election.cc:343] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 8af99e0bd8774a678575847a9ec2e557
I20251214 20:52:20.110966 713 leader_election.cc:304] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:20.111343 1962 raft_consensus.cc:3060] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.111405 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 24ff58f321124d12b5330dca185d426e (table default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]) successfully deleted
I20251214 20:52:20.114531 1962 raft_consensus.cc:2749] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.116010 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b4334a1d99a54c2eabe2fb909563313c" } modify_external_catalogs: true
I20251214 20:52:20.116267 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b4334a1d99a54c2eabe2fb909563313c" } modify_external_catalogs: true
I20251214 20:52:20.117519 1977 tablet_replica.cc:333] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.117910 1977 raft_consensus.cc:2243] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.118363 1982 tablet_replica.cc:333] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:20.118770 1977 raft_consensus.cc:2272] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.119663 1977 ts_tablet_manager.cc:1916] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.119659 1982 raft_consensus.cc:2243] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.119971 1982 raft_consensus.cc:2272] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.120657 1982 ts_tablet_manager.cc:1916] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.121419 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_725af8309e494ad3a5a40a73d2f585fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.122066 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_725af8309e494ad3a5a40a73d2f585fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.138388 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 65e79db74fa046698507c83416ee1669
I20251214 20:52:20.140185 1977 ts_tablet_manager.cc:1929] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:20.140833 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 65e79db74fa046698507c83416ee1669 with delete_type TABLET_DATA_DELETED (replaced by tablet 4dcd511a5dca4a2cbda81fbc255ce354) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.142963 1856 ts_tablet_manager.cc:1929] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.143327 451 catalog_manager.cc:6724] ReplaceTablet: tablet 65e79db74fa046698507c83416ee1669 deleted and replaced by tablet 4dcd511a5dca4a2cbda81fbc255ce354
I20251214 20:52:20.143285 1856 log.cc:1199] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/f0d661a1bb034053a4634ff537eafc7d
I20251214 20:52:20.144157 1856 ts_tablet_manager.cc:1950] T f0d661a1bb034053a4634ff537eafc7d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.141587 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 65e79db74fa046698507c83416ee1669 with delete_type TABLET_DATA_DELETED (replaced by tablet 4dcd511a5dca4a2cbda81fbc255ce354) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.144824 1977 log.cc:1199] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/3bbb28259c1442ae94f4eff76c25c539
I20251214 20:52:20.145367 1977 ts_tablet_manager.cc:1950] T 3bbb28259c1442ae94f4eff76c25c539 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.145483 1102 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:20.146761 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 3bbb28259c1442ae94f4eff76c25c539 (table default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]) successfully deleted
I20251214 20:52:20.147037 1206 tablet_bootstrap.cc:492] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:20.148365 1977 tablet_replica.cc:333] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.155411 1977 raft_consensus.cc:2243] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.155843 1977 raft_consensus.cc:2272] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.148201 1908 tablet.cc:1722] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:20.156700 1977 ts_tablet_manager.cc:1916] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.151948 1963 raft_consensus.cc:2749] T cf04f91e79754569a51d468ab576b0dc P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.157712 1206 ts_tablet_manager.cc:1403] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.354s user 0.011s sys 0.002s
I20251214 20:52:20.158756 1206 raft_consensus.cc:359] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.161291 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 36) took 1398 ms. Trace:
W20251214 20:52:20.159600 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 48892.2us
I20251214 20:52:20.161456 1856 rpcz_store.cc:276] 1214 20:52:18.762308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.764028 (+ 1720us) service_pool.cc:225] Handling call
1214 20:52:20.102222 (+1338194us) ts_tablet_manager.cc:1151] Deleting tablet f0d661a1bb034053a4634ff537eafc7d
1214 20:52:20.102234 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.122385 (+ 20151us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.140564 (+ 18179us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.161214 (+ 20650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299822}
I20251214 20:52:20.162144 1856 tablet_replica.cc:333] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:20.162335 989 leader_election.cc:343] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.162508 1856 raft_consensus.cc:2243] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.162976 1856 raft_consensus.cc:2272] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.149482 450 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]). Expected version 1 got 0
I20251214 20:52:20.164093 450 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_55fe32e16f364e58afdd57c80a8966ae [id=6157817e96d642d09f9941ad7f7e82ee]). Expected version 1 got 0
I20251214 20:52:20.164501 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 40a9a6192c744282ab5672cbe15fd299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.164855 750 tablet_service.cc:1552] Processing DeleteTablet for tablet ba50e32f7e724868bb891045a939e362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.165439 1856 ts_tablet_manager.cc:1916] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.164692 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet f0d661a1bb034053a4634ff537eafc7d (table default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90]) successfully deleted
I20251214 20:52:20.168053 450 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 3fa0317e999b483788e5c5652e6e83de (table default.Table_6c8f53a62dca43e2b8189d4a1fda9989 [id=ef22180b36a241fea13d0e622c02b50a]). Expected version 2 got 1
W20251214 20:52:20.172144 991 consensus_peers.cc:597] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe4086fc7a4f4db0b7541f06c9833aab. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:20.172681 450 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:20.173061 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_431bf74dfb32400bb0b2ef098c18e8df" } modify_external_catalogs: true
I20251214 20:52:20.173312 450 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:20.173381 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_431bf74dfb32400bb0b2ef098c18e8df" } modify_external_catalogs: true
I20251214 20:52:20.174036 450 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:20.174412 450 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 02e256d4537249a394b80dc36beefef4 (table default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]). Expected version 1 got 0
I20251214 20:52:20.175243 1856 ts_tablet_manager.cc:1929] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.177138 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_55706259657e4a09aa794297547e9d98" } new_table_name: "default.Table_7660878c81e04c17b9b6fa8d61ef43a6" modify_external_catalogs: true
I20251214 20:52:20.182317 1993 tablet.cc:1722] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:20.182729 1856 log.cc:1199] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/2da66ec83e9140f7a605f0c86abff468
I20251214 20:52:20.181572 1898 tablet.cc:1722] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:20.183329 1856 ts_tablet_manager.cc:1950] T 2da66ec83e9140f7a605f0c86abff468 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.185058 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.185740 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 40) took 1422 ms. Trace:
I20251214 20:52:20.160916 1206 raft_consensus.cc:385] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.185895 1206 raft_consensus.cc:740] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.186048 1856 rpcz_store.cc:276] 1214 20:52:18.762763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.762825 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:20.162021 (+1399196us) ts_tablet_manager.cc:1151] Deleting tablet 2da66ec83e9140f7a605f0c86abff468
1214 20:52:20.162033 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.168916 (+ 6883us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.175077 (+ 6161us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.185665 (+ 10588us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1356723}
I20251214 20:52:20.186713 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 2da66ec83e9140f7a605f0c86abff468 (table default.Table_2325a2c5e97f45ebac4c943380bb3525 [id=0c6d9651f90f479293852cb3fb1d320f]) successfully deleted
I20251214 20:52:20.180842 1667 consensus_queue.cc:1048] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:20.188220 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 42) took 1425 ms. Trace:
W20251214 20:52:20.168560 447 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 98b7cac3930a4a9eb16ac413c8f1e092 (table default.Table_e5f3db562155448e91e2e4314254cbe6 [id=826a2666d167435c8fdfd9ab8b8819e6], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 98b7cac3930a4a9eb16ac413c8f1e092 consensus state has no leader
I20251214 20:52:20.188329 1856 rpcz_store.cc:276] 1214 20:52:18.763061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.763115 (+ 54us) service_pool.cc:225] Handling call
1214 20:52:20.187384 (+1424269us) ts_tablet_manager.cc:1151] Deleting tablet d6c0f445cd7e44769d79ad5566fd8d97
1214 20:52:20.187395 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.188133 (+ 738us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1381894}
W20251214 20:52:20.188866 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet d6c0f445cd7e44769d79ad5566fd8d97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6c0f445cd7e44769d79ad5566fd8d97 already in progress: creating tablet
I20251214 20:52:20.184623 1977 ts_tablet_manager.cc:1929] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:20.189201 1977 log.cc:1199] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/d6c0f445cd7e44769d79ad5566fd8d97
I20251214 20:52:20.189378 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 35) took 1428 ms. Trace:
I20251214 20:52:20.189476 1856 rpcz_store.cc:276] 1214 20:52:18.761278 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.763252 (+ 1974us) service_pool.cc:225] Handling call
1214 20:52:20.189157 (+1425905us) ts_tablet_manager.cc:1151] Deleting tablet 7fcfcfe81fe24d369eea696efc61d21c
1214 20:52:20.189167 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.189320 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1383538}
I20251214 20:52:20.189656 1977 ts_tablet_manager.cc:1950] T d6c0f445cd7e44769d79ad5566fd8d97 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
W20251214 20:52:20.189925 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 7fcfcfe81fe24d369eea696efc61d21c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fcfcfe81fe24d369eea696efc61d21c already in progress: creating tablet
I20251214 20:52:20.190663 1977 tablet_replica.cc:333] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.190896 1977 raft_consensus.cc:2243] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.191085 1977 raft_consensus.cc:2272] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.192342 1977 ts_tablet_manager.cc:1916] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.192932 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet d6c0f445cd7e44769d79ad5566fd8d97 (table default.Table_d863e40d2a9e46649f9dc97f33a616db [id=cd65155def8244e59f08dac4d695567d]) successfully deleted
W20251214 20:52:20.196249 447 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:39607 (request call id 150) took 2513 ms (client timeout 1000 ms). Trace:
W20251214 20:52:20.196403 447 rpcz_store.cc:269] 1214 20:52:17.682908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:17.864863 (+181955us) service_pool.cc:225] Handling call
1214 20:52:17.928423 (+ 63560us) spinlock_profiling.cc:243] Waited 59 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b17aa08 00007ff411c87ca6 00007ff411c879d5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:20.196177 (+2267754us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":135,"spinlock_wait_cycles":135708032}
I20251214 20:52:20.197535 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 37) took 1435 ms. Trace:
I20251214 20:52:20.197705 1856 rpcz_store.cc:276] 1214 20:52:18.762444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.762504 (+ 60us) service_pool.cc:225] Handling call
1214 20:52:20.196536 (+1434032us) ts_tablet_manager.cc:1151] Deleting tablet bb698ff30fcc47d09000cc68d2c5225b
1214 20:52:20.196546 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.197464 (+ 918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1390743}
W20251214 20:52:20.198196 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet bb698ff30fcc47d09000cc68d2c5225b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb698ff30fcc47d09000cc68d2c5225b already in progress: creating tablet
I20251214 20:52:20.198350 1856 tablet_replica.cc:333] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.198626 1856 raft_consensus.cc:2243] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.198925 1856 raft_consensus.cc:2272] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.200332 1908 tablet.cc:1722] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251214 20:52:20.201592 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d01db63fd5d2474a9defdce0b98fc29d (table default.Table_13f016135de44cf78187fbc4f9f8c622 [id=fc160acc971e4f7d8290560ee8aab065], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d01db63fd5d2474a9defdce0b98fc29d consensus state has no leader
I20251214 20:52:20.202200 1202 tablet_bootstrap.cc:492] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:20.202471 1202 ts_tablet_manager.cc:1403] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.306s user 0.007s sys 0.002s
I20251214 20:52:20.199609 1203 tablet_bootstrap.cc:492] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:20.203161 1203 ts_tablet_manager.cc:1403] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.142s user 0.005s sys 0.034s
I20251214 20:52:20.204047 1203 raft_consensus.cc:359] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.204268 1203 raft_consensus.cc:385] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.204348 1203 raft_consensus.cc:740] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.204632 1203 consensus_queue.cc:260] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.205719 1203 ts_tablet_manager.cc:1434] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:20.206228 1203 tablet_bootstrap.cc:492] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:20.208185 1206 consensus_queue.cc:260] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.208839 1203 tablet_bootstrap.cc:654] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.209265 1206 ts_tablet_manager.cc:1434] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.051s user 0.001s sys 0.001s
I20251214 20:52:20.209710 1206 tablet_bootstrap.cc:492] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:20.210640 1856 ts_tablet_manager.cc:1916] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.211822 1206 tablet_bootstrap.cc:654] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.213212 1977 ts_tablet_manager.cc:1929] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.213613 1977 log.cc:1199] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/b652a8a80246443491551e76e68d2a8b
I20251214 20:52:20.214668 1977 ts_tablet_manager.cc:1950] T b652a8a80246443491551e76e68d2a8b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.217777 1977 tablet_replica.cc:333] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.218135 1856 ts_tablet_manager.cc:1929] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.218142 1977 raft_consensus.cc:2243] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.218367 1977 raft_consensus.cc:2272] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.218448 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet b652a8a80246443491551e76e68d2a8b (table default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]) successfully deleted
W20251214 20:52:20.218828 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8d74d3abc45a41969289c0a0892bfa04 because the tablet was not found. No further retry: Not found: Tablet not found: 8d74d3abc45a41969289c0a0892bfa04
I20251214 20:52:20.219031 1977 ts_tablet_manager.cc:1916] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.223564 1667 consensus_queue.cc:1048] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:20.231738 1856 log.cc:1199] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/c25832da5c1644329e18653000f3330e
I20251214 20:52:20.232286 1977 ts_tablet_manager.cc:1929] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.232517 1856 ts_tablet_manager.cc:1950] T c25832da5c1644329e18653000f3330e P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.233464 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 45) took 1446 ms. Trace:
I20251214 20:52:20.233610 1856 rpcz_store.cc:276] 1214 20:52:18.786932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.787025 (+ 93us) service_pool.cc:225] Handling call
1214 20:52:20.198245 (+1411220us) ts_tablet_manager.cc:1151] Deleting tablet c25832da5c1644329e18653000f3330e
1214 20:52:20.198265 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.214978 (+ 16713us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.217990 (+ 3012us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.233422 (+ 15432us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1392292}
I20251214 20:52:20.235749 1977 log.cc:1199] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/f3a2143b5508445492b970d2588cf1cf
I20251214 20:52:20.236176 1856 tablet_replica.cc:333] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.236860 1977 ts_tablet_manager.cc:1950] T f3a2143b5508445492b970d2588cf1cf P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.236702 1856 raft_consensus.cc:2243] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.238420 1856 raft_consensus.cc:2272] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.239542 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet f3a2143b5508445492b970d2588cf1cf (table default.Table_d653f9c533784976ac955f90dc39eb56 [id=295a34df92b44b738e3aa3d1d8195673]) successfully deleted
I20251214 20:52:20.239661 1977 tablet_replica.cc:333] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.239933 1977 raft_consensus.cc:2243] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.243467 1977 raft_consensus.cc:2272] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.243608 1856 ts_tablet_manager.cc:1916] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.244464 1977 ts_tablet_manager.cc:1916] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.255667 1977 ts_tablet_manager.cc:1929] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.262943 1856 ts_tablet_manager.cc:1929] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.263624 1977 log.cc:1199] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/a0279992d1f2419ab1caae116a0391b2
I20251214 20:52:20.264055 1856 log.cc:1199] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/9523d72fd91445539d67e1eac2a80afd
I20251214 20:52:20.264302 1977 ts_tablet_manager.cc:1950] T a0279992d1f2419ab1caae116a0391b2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.264660 1856 ts_tablet_manager.cc:1950] T 9523d72fd91445539d67e1eac2a80afd P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.265462 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 34) took 1508 ms. Trace:
I20251214 20:52:20.265566 1856 rpcz_store.cc:276] 1214 20:52:18.757311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.757364 (+ 53us) service_pool.cc:225] Handling call
1214 20:52:20.236092 (+1478728us) ts_tablet_manager.cc:1151] Deleting tablet 9523d72fd91445539d67e1eac2a80afd
1214 20:52:20.236107 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.255786 (+ 19679us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.262729 (+ 6943us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.265402 (+ 2673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429026}
I20251214 20:52:20.265720 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet a0279992d1f2419ab1caae116a0391b2 (table default.Table_4e46507057c94aa7a347491bf2e482ff [id=3906806ef8294342a79b9face8753212]) successfully deleted
I20251214 20:52:20.266099 1977 tablet_replica.cc:333] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.266408 1977 raft_consensus.cc:2243] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.266903 1977 raft_consensus.cc:2272] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.267838 1977 ts_tablet_manager.cc:1916] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.270949 1856 tablet_replica.cc:333] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.271517 1856 raft_consensus.cc:2243] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.272195 1856 raft_consensus.cc:2272] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.273217 1856 ts_tablet_manager.cc:1916] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.288149 1977 ts_tablet_manager.cc:1929] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.288905 1650 raft_consensus.cc:493] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.293269 1650 raft_consensus.cc:515] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.294224 1650 leader_election.cc:290] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:20.298143 1977 log.cc:1199] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/c16895ca08d94b56b5c4bc9352e330ef
I20251214 20:52:20.298841 1977 ts_tablet_manager.cc:1950] T c16895ca08d94b56b5c4bc9352e330ef P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.298859 1856 ts_tablet_manager.cc:1929] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.300287 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet c16895ca08d94b56b5c4bc9352e330ef (table default.Table_9f6bb794724346069627cd2f14a5cd1b [id=4e30619d28324239bc069a5ce053b167]) successfully deleted
I20251214 20:52:20.303522 1856 log.cc:1199] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/b652a8a80246443491551e76e68d2a8b
I20251214 20:52:20.304385 1856 ts_tablet_manager.cc:1950] T b652a8a80246443491551e76e68d2a8b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.305852 1977 tablet_replica.cc:333] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.306939 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 44) took 1522 ms. Trace:
I20251214 20:52:20.306959 1977 raft_consensus.cc:2243] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.307435 1977 pending_rounds.cc:70] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:20.307559 1977 pending_rounds.cc:77] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493722848440320 op_type: NO_OP noop_request { }
I20251214 20:52:20.307730 1977 raft_consensus.cc:2889] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:20.307881 1977 raft_consensus.cc:2272] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.307281 1856 rpcz_store.cc:276] 1214 20:52:18.783496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.783671 (+ 175us) service_pool.cc:225] Handling call
1214 20:52:20.270844 (+1487173us) ts_tablet_manager.cc:1151] Deleting tablet b652a8a80246443491551e76e68d2a8b
1214 20:52:20.270856 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.290651 (+ 19795us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.298648 (+ 7997us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.305911 (+ 7263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1459979}
I20251214 20:52:20.308633 1977 ts_tablet_manager.cc:1916] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.310420 1856 tablet_replica.cc:333] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.310598 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 25974e5db555447baf5799775d15cd14
I20251214 20:52:20.310786 1856 raft_consensus.cc:2243] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.311167 1856 raft_consensus.cc:2272] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.312016 455 catalog_manager.cc:6724] ReplaceTablet: tablet 25974e5db555447baf5799775d15cd14 deleted and replaced by tablet d171301941fa43908e05984c3b016007
I20251214 20:52:20.312222 455 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 104) took 1192 ms. Trace:
I20251214 20:52:20.312429 455 rpcz_store.cc:276] 1214 20:52:19.119547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.021335 (+901788us) service_pool.cc:225] Handling call
1214 20:52:20.312187 (+290852us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:20.279438 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.279604 (+ 166us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.279611 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:20.279613 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:20.279746 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.279761 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.279767 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:20.279877 (+ 110us) tablet.cc:699] Row locks acquired
1214 20:52:20.279880 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:20.279927 (+ 47us) write_op.cc:270] Start()
1214 20:52:20.279973 (+ 46us) write_op.cc:276] Timestamp: P: 1765745540279839 usec, L: 0
1214 20:52:20.279978 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.280202 (+ 224us) log.cc:844] Serialized 656 byte log entry
1214 20:52:20.308871 (+ 28669us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.309789 (+ 918us) write_op.cc:301] APPLY: starting
1214 20:52:20.309800 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.309831 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.309852 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.309958 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.310014 (+ 56us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:20.310020 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:20.310080 (+ 60us) log.cc:844] Serialized 29 byte log entry
1214 20:52:20.310293 (+ 213us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.310309 (+ 16us) write_op.cc:454] Released schema lock
1214 20:52:20.310315 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":882,"num_ops":2,"prepare.queue_time_us":936,"prepare.run_cpu_time_us":840,"prepare.run_wall_time_us":839,"replication_time_us":28873}]]}
I20251214 20:52:20.314069 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 25974e5db555447baf5799775d15cd14 with delete_type TABLET_DATA_DELETED (replaced by tablet d171301941fa43908e05984c3b016007) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.315085 1202 raft_consensus.cc:359] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.315474 1202 raft_consensus.cc:385] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.315621 1202 raft_consensus.cc:740] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.315683 1928 consensus_queue.cc:1048] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:20.316038 1202 consensus_queue.cc:260] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.316722 1202 ts_tablet_manager.cc:1434] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.114s user 0.001s sys 0.001s
I20251214 20:52:20.317168 1202 tablet_bootstrap.cc:492] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:20.316758 1856 ts_tablet_manager.cc:1916] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.197990 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_e26cdfc4ab8f4f19b52de34448bd70e5" } modify_external_catalogs: true
I20251214 20:52:20.323961 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fcfcfe81fe24d369eea696efc61d21c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:20.325946 449 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 65e79db74fa046698507c83416ee1669 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 65e79db74fa046698507c83416ee1669 consensus state has no leader
I20251214 20:52:20.319419 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_e26cdfc4ab8f4f19b52de34448bd70e5" } modify_external_catalogs: true
W20251214 20:52:20.334048 851 leader_election.cc:343] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.335191 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_30e486c0b499465c89d73644d7980f81" } modify_external_catalogs: true
I20251214 20:52:20.335566 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_30e486c0b499465c89d73644d7980f81" } modify_external_catalogs: true
I20251214 20:52:20.341202 1977 ts_tablet_manager.cc:1929] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.341475 1977 log.cc:1199] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/c2ae9dc8ead8403cb7215af8778872f0
I20251214 20:52:20.342140 1977 ts_tablet_manager.cc:1950] T c2ae9dc8ead8403cb7215af8778872f0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.343734 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet c2ae9dc8ead8403cb7215af8778872f0 (table default.Table_2342c4af4f9b420b9b7cbcf327ce1733 [id=976ca3fb3a164669abeff0d8eb137e41]) successfully deleted
I20251214 20:52:20.347703 1977 tablet_replica.cc:333] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.348105 1977 raft_consensus.cc:2243] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.348369 1856 ts_tablet_manager.cc:1929] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.348443 1977 raft_consensus.cc:2272] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.352233 1977 ts_tablet_manager.cc:1916] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.353135 1856 log.cc:1199] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/6b4f582d8165439d86d9228c146dc41a
I20251214 20:52:20.353852 1856 ts_tablet_manager.cc:1950] T 6b4f582d8165439d86d9228c146dc41a P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.355641 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 46) took 1544 ms. Trace:
I20251214 20:52:20.356983 1856 rpcz_store.cc:276] 1214 20:52:18.810969 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.811235 (+ 266us) service_pool.cc:225] Handling call
1214 20:52:20.308393 (+1497158us) ts_tablet_manager.cc:1151] Deleting tablet 6b4f582d8165439d86d9228c146dc41a
1214 20:52:20.308405 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.322964 (+ 14559us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.348183 (+ 25219us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.355571 (+ 7388us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43,"tablet-delete.queue_time_us":1486092}
I20251214 20:52:20.358170 1203 tablet_bootstrap.cc:492] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:20.358659 1203 ts_tablet_manager.cc:1403] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.153s user 0.022s sys 0.022s
I20251214 20:52:20.360029 1203 raft_consensus.cc:359] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:20.360383 1203 raft_consensus.cc:385] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.360512 1203 raft_consensus.cc:740] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.360867 1203 consensus_queue.cc:260] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:20.361521 1203 ts_tablet_manager.cc:1434] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251214 20:52:20.363082 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:34819, user_credentials={real_user=slave}} blocked reactor thread for 96775.3us
I20251214 20:52:20.363687 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet c25832da5c1644329e18653000f3330e (table default.Table_c5d8c1bdf3344ec5a0d409a1abf98b45 [id=5427e484a5a94314a13be3f387c06d64]) successfully deleted
I20251214 20:52:20.364452 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 9523d72fd91445539d67e1eac2a80afd (table default.Table_76a2939ccdea4fdeaa1da24713bc3ca7 [id=edb7e58d7c0b4ddea05ba0139f321042]) successfully deleted
I20251214 20:52:20.364789 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet b652a8a80246443491551e76e68d2a8b (table default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]) successfully deleted
I20251214 20:52:20.365130 1203 tablet_bootstrap.cc:492] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:20.366361 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 50) took 1542 ms. Trace:
I20251214 20:52:20.366580 1856 rpcz_store.cc:276] 1214 20:52:18.824300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.824365 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:20.366134 (+1541769us) ts_tablet_manager.cc:1151] Deleting tablet 4d6e7306254044449f1127fb963ca5f0
1214 20:52:20.366144 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.366287 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1539086}
I20251214 20:52:20.367086 1856 tablet_replica.cc:333] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.367408 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 6b4f582d8165439d86d9228c146dc41a (table default.Table_22df18c031ab4778ae90eb3580e9b8b7 [id=3f105cc6dd6944ceb9783a1ebe923d11]) successfully deleted
I20251214 20:52:20.369261 1856 raft_consensus.cc:2243] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.215879 739 tablet_service.cc:1552] Processing DeleteTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f with delete_type TABLET_DATA_DELETED (replaced by tablet f11834cabdad445b8e53cd237d146282) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.371657 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 06381efe75284e649fd6d1a9903cfa9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.371711 1856 raft_consensus.cc:2272] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.371892 739 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.372051 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d6e7306254044449f1127fb963ca5f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.372182 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 69f6108792fa4c2a9744e3248c76251f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.372335 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d89ee602c524c96a21c0a235af07481 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e11723c042a4637839f1b13bed0935a) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.216398 756 tablet_service.cc:1552] Processing DeleteTablet for tablet ee7d709316b24b8eb223a838710c2979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.372431 1203 tablet_bootstrap.cc:654] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.220402 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7c6fe3b24daca70711a1990af556 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.220834 745 tablet_service.cc:1552] Processing DeleteTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.221290 747 tablet_service.cc:1552] Processing DeleteTablet for tablet b1df0dad2fb040a5a12e0401e00c87db with delete_type TABLET_DATA_DELETED (replaced by tablet 8fcdd6bafa5c4aa6b59668cde523d010) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.223737 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 65e79db74fa046698507c83416ee1669 with delete_type TABLET_DATA_DELETED (replaced by tablet 4dcd511a5dca4a2cbda81fbc255ce354) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.224046 750 tablet_service.cc:1552] Processing DeleteTablet for tablet ec85857f047e44d78835305ddcb70768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.224201 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d74d3abc45a41969289c0a0892bfa04 with delete_type TABLET_DATA_DELETED (replaced by tablet 727e7b61ae3942098712720d344c5310) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.224524 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 012b81223d0f4ae9b742014e329f11a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.225414 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.225718 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a10663bd306d40e7a6375d06d69be22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.226142 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b1a937150ce4ecfa210c8465d682496 with delete_type TABLET_DATA_DELETED (replaced by tablet f1eb4c3aa654480caaa69dbd5047365c) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.226465 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff58f321124d12b5330dca185d426e with delete_type TABLET_DATA_DELETED (replaced by tablet 42bdcecba5534294aca1eb419f18e9d2) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.226763 749 tablet_service.cc:1552] Processing DeleteTablet for tablet a0279992d1f2419ab1caae116a0391b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.227072 744 tablet_service.cc:1552] Processing DeleteTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.227470 740 tablet_service.cc:1552] Processing DeleteTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf with delete_type TABLET_DATA_DELETED (replaced by tablet 6fef1d6add764fd28a8f4995b337ea71) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.297194 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fcfcfe81fe24d369eea696efc61d21c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.373925 1856 ts_tablet_manager.cc:1916] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.368207 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:20.374418 848 leader_election.cc:343] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.374574 848 leader_election.cc:304] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:20.375046 1762 raft_consensus.cc:2749] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.368615 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 649f5bd67cbb4f61b7066269f0bf9266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.368813 757 tablet_service.cc:1552] Processing DeleteTablet for tablet c16895ca08d94b56b5c4bc9352e330ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.369028 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 25974e5db555447baf5799775d15cd14 with delete_type TABLET_DATA_DELETED (replaced by tablet d171301941fa43908e05984c3b016007) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.381742 1977 ts_tablet_manager.cc:1929] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:20.381943 1977 log.cc:1199] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/a34eb8fa3e34427495fa5985b7ba81bf
I20251214 20:52:20.382328 1977 ts_tablet_manager.cc:1950] T a34eb8fa3e34427495fa5985b7ba81bf P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.384255 1915 raft_consensus.cc:493] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.384547 1915 raft_consensus.cc:515] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.385731 1915 leader_election.cc:290] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:20.386680 991 leader_election.cc:343] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: d6c0f445cd7e44769d79ad5566fd8d97
I20251214 20:52:20.387326 1982 ts_tablet_manager.cc:1929] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:20.387477 1982 log.cc:1199] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/fe4086fc7a4f4db0b7541f06c9833aab
I20251214 20:52:20.387883 1982 ts_tablet_manager.cc:1950] T fe4086fc7a4f4db0b7541f06c9833aab P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:20.389966 1982 tablet_replica.cc:333] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:20.390202 1982 raft_consensus.cc:2243] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.390388 1982 raft_consensus.cc:2272] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.390962 1982 ts_tablet_manager.cc:1916] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.393134 1915 raft_consensus.cc:493] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.393302 1915 raft_consensus.cc:515] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.394194 1915 leader_election.cc:290] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.394763 774 raft_consensus.cc:3060] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:20.395078 991 leader_election.cc:343] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 324379e3217442c3addcfe58060b9ad7
I20251214 20:52:20.395718 1977 tablet_replica.cc:333] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.396025 1977 raft_consensus.cc:2243] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.390440 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet a34eb8fa3e34427495fa5985b7ba81bf (table default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]) successfully deleted
I20251214 20:52:20.392855 455 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 760bdc4e589f459cbf811ad54d4efeec
I20251214 20:52:20.396386 1203 tablet_bootstrap.cc:492] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:20.396559 1203 ts_tablet_manager.cc:1403] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20251214 20:52:20.396836 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d7519f28730149dc956626fb163db031" } new_table_name: "default.Table_4a58233140a046368357556e00275995" modify_external_catalogs: true
I20251214 20:52:20.396306 1977 raft_consensus.cc:2272] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.397390 1203 raft_consensus.cc:359] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.397605 1203 raft_consensus.cc:385] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.397687 1203 raft_consensus.cc:740] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.397965 1203 consensus_queue.cc:260] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:20.398288 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 4d6e7306254044449f1127fb963ca5f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d6e7306254044449f1127fb963ca5f0 already in progress: creating tablet
I20251214 20:52:20.398491 1203 ts_tablet_manager.cc:1434] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:20.399004 1977 ts_tablet_manager.cc:1916] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.399601 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 25974e5db555447baf5799775d15cd14 with delete_type TABLET_DATA_DELETED (replaced by tablet d171301941fa43908e05984c3b016007) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.402486 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "324379e3217442c3addcfe58060b9ad7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.403326 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6c0f445cd7e44769d79ad5566fd8d97" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.404757 1203 tablet_bootstrap.cc:492] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:20.407845 1203 tablet_bootstrap.cc:654] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.417122 888 tablet_service.cc:1505] Processing CreateTablet for tablet 3915abc53b084854912a8e4119eefc7e (DEFAULT_TABLE table=default.Table_ae7fe3d1672c46baa454f28810e589af [id=f3cf4205b86e4413aef210d319ed868b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.420291 1915 consensus_queue.cc:1048] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:20.420778 449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:39607 (request call id 151) took 1540 ms (client timeout 1000 ms). Trace:
I20251214 20:52:20.421674 1203 tablet_bootstrap.cc:492] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:20.421711 884 tablet_service.cc:1505] Processing CreateTablet for tablet db0ad1aa4193467db74cd707e7084b71 (DEFAULT_TABLE table=default.Table_3b13fb90bb2c4a0db268b7777b08f240 [id=8acc06886693413682b246f25970fa5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.421849 1203 ts_tablet_manager.cc:1403] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20251214 20:52:20.421896 892 tablet_service.cc:1505] Processing CreateTablet for tablet 8eb1006b0ba440dcba2f9de04dca2702 (DEFAULT_TABLE table=default.Table_c2048db412744e2da280a49ec3d679b8 [id=e49aebe4f8214b9c879993f8cd2a1f6a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:20.421005 449 rpcz_store.cc:269] 1214 20:52:18.880276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.074812 (+194536us) service_pool.cc:225] Handling call
1214 20:52:20.420714 (+1345902us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38482,"reported_tablets":151}
I20251214 20:52:20.422317 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3915abc53b084854912a8e4119eefc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.422704 1203 raft_consensus.cc:359] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:20.422820 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0ad1aa4193467db74cd707e7084b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.422890 1203 raft_consensus.cc:385] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.422982 1203 raft_consensus.cc:740] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.423285 1203 consensus_queue.cc:260] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:20.423394 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eb1006b0ba440dcba2f9de04dca2702. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.423982 1203 ts_tablet_manager.cc:1434] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:20.424913 1203 tablet_bootstrap.cc:492] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:20.427378 878 tablet_service.cc:1505] Processing CreateTablet for tablet 4e4dc66681ef48479ae3cb95b091564a (DEFAULT_TABLE table=default.Table_00ec2fefc3d84a8b9f94bf9d194e2045 [id=12e7c17407a444dd8e0c0ad40fb89e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.427863 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e4dc66681ef48479ae3cb95b091564a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.428587 1203 tablet_bootstrap.cc:654] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.431360 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet fe4086fc7a4f4db0b7541f06c9833aab (table default.Table_8076bc8b31ba411faf089f314e6ed2ef [id=25016f04bb864f92a1cdfb6ceaa9fba9]) successfully deleted
W20251214 20:52:20.431922 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 7dd657f3baff40bb8cb7f4648367f7a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dd657f3baff40bb8cb7f4648367f7a1 already in progress: creating tablet
W20251214 20:52:20.432380 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 98b7cac3930a4a9eb16ac413c8f1e092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98b7cac3930a4a9eb16ac413c8f1e092 already in progress: creating tablet
W20251214 20:52:20.432883 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3c3962c33a294fa98f8bd485268aee71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c3962c33a294fa98f8bd485268aee71 already in progress: creating tablet
W20251214 20:52:20.433403 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3bbb28259c1442ae94f4eff76c25c539 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bbb28259c1442ae94f4eff76c25c539 already in progress: creating tablet
I20251214 20:52:20.433818 1915 raft_consensus.cc:493] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.434015 1915 raft_consensus.cc:515] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.434904 1915 leader_election.cc:290] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:20.435501 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c9b7149a436a4cab80b39adccf8567f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b7149a436a4cab80b39adccf8567f9 already in progress: creating tablet
W20251214 20:52:20.435812 991 leader_election.cc:343] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 24ff58f321124d12b5330dca185d426e
W20251214 20:52:20.436178 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 9c0f49e5d78240a296e47eccb3218e4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c0f49e5d78240a296e47eccb3218e4f already in progress: creating tablet
I20251214 20:52:20.436228 1202 tablet_bootstrap.cc:654] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:20.436901 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c2ae9dc8ead8403cb7215af8778872f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2ae9dc8ead8403cb7215af8778872f0 already in progress: creating tablet
W20251214 20:52:20.439388 989 consensus_peers.cc:597] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:20.440058 1977 ts_tablet_manager.cc:1929] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.440711 1915 consensus_queue.cc:1048] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:20.441087 1977 log.cc:1199] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/40a9a6192c744282ab5672cbe15fd299
I20251214 20:52:20.441639 1977 ts_tablet_manager.cc:1950] T 40a9a6192c744282ab5672cbe15fd299 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
W20251214 20:52:20.441642 989 leader_election.cc:343] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.441809 989 leader_election.cc:304] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:20.442252 1915 raft_consensus.cc:2749] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:20.443789 989 leader_election.cc:343] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.443935 989 leader_election.cc:304] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:20.444329 1915 raft_consensus.cc:2749] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.445012 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 40a9a6192c744282ab5672cbe15fd299 (table default.Table_847906e708bd41e9b9507efc8f52347f [id=4afcd94192854e049c21f8e6852cd56f]) successfully deleted
I20251214 20:52:20.445451 1977 tablet_replica.cc:333] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.447387 1977 raft_consensus.cc:2243] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.447788 1977 raft_consensus.cc:2272] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.447850 738 tablet_service.cc:1505] Processing CreateTablet for tablet 3915abc53b084854912a8e4119eefc7e (DEFAULT_TABLE table=default.Table_ae7fe3d1672c46baa454f28810e589af [id=f3cf4205b86e4413aef210d319ed868b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.448065 750 tablet_service.cc:1505] Processing CreateTablet for tablet 4e4dc66681ef48479ae3cb95b091564a (DEFAULT_TABLE table=default.Table_00ec2fefc3d84a8b9f94bf9d194e2045 [id=12e7c17407a444dd8e0c0ad40fb89e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.448451 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3915abc53b084854912a8e4119eefc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.448715 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e4dc66681ef48479ae3cb95b091564a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.448851 1977 ts_tablet_manager.cc:1916] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.451521 757 tablet_service.cc:1505] Processing CreateTablet for tablet 8eb1006b0ba440dcba2f9de04dca2702 (DEFAULT_TABLE table=default.Table_c2048db412744e2da280a49ec3d679b8 [id=e49aebe4f8214b9c879993f8cd2a1f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.451929 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eb1006b0ba440dcba2f9de04dca2702. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.454501 743 tablet_service.cc:1505] Processing CreateTablet for tablet db0ad1aa4193467db74cd707e7084b71 (DEFAULT_TABLE table=default.Table_3b13fb90bb2c4a0db268b7777b08f240 [id=8acc06886693413682b246f25970fa5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.457222 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0ad1aa4193467db74cd707e7084b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.457445 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24ff58f321124d12b5330dca185d426e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.458345 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a218538b99ae4cf6b55add7cf3b995a4" } new_table_name: "default.Table_d2beb8dc1b054cada5e4c2ec9fabc8d4" modify_external_catalogs: true
I20251214 20:52:20.460206 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 8eb1006b0ba440dcba2f9de04dca2702 (DEFAULT_TABLE table=default.Table_c2048db412744e2da280a49ec3d679b8 [id=e49aebe4f8214b9c879993f8cd2a1f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.460325 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 3915abc53b084854912a8e4119eefc7e (DEFAULT_TABLE table=default.Table_ae7fe3d1672c46baa454f28810e589af [id=f3cf4205b86e4413aef210d319ed868b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.460726 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eb1006b0ba440dcba2f9de04dca2702. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.461038 1034 tablet_service.cc:1505] Processing CreateTablet for tablet db0ad1aa4193467db74cd707e7084b71 (DEFAULT_TABLE table=default.Table_3b13fb90bb2c4a0db268b7777b08f240 [id=8acc06886693413682b246f25970fa5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.461452 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db0ad1aa4193467db74cd707e7084b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.460208 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 4e4dc66681ef48479ae3cb95b091564a (DEFAULT_TABLE table=default.Table_00ec2fefc3d84a8b9f94bf9d194e2045 [id=12e7c17407a444dd8e0c0ad40fb89e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.463685 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e4dc66681ef48479ae3cb95b091564a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.463737 1977 ts_tablet_manager.cc:1929] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.463913 1977 log.cc:1199] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/9523d72fd91445539d67e1eac2a80afd
I20251214 20:52:20.464380 1977 ts_tablet_manager.cc:1950] T 9523d72fd91445539d67e1eac2a80afd P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.465713 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 9523d72fd91445539d67e1eac2a80afd (table default.Table_76a2939ccdea4fdeaa1da24713bc3ca7 [id=edb7e58d7c0b4ddea05ba0139f321042]) successfully deleted
W20251214 20:52:20.466190 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 59338.6us
I20251214 20:52:20.467319 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3915abc53b084854912a8e4119eefc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.384256 1856 ts_tablet_manager.cc:1929] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.472976 1856 log.cc:1199] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/2842f1dc0c374d24946f90dac1cf8aa5
I20251214 20:52:20.473697 1856 ts_tablet_manager.cc:1950] T 2842f1dc0c374d24946f90dac1cf8aa5 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.474671 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 47) took 1655 ms. Trace:
I20251214 20:52:20.475265 1856 rpcz_store.cc:276] 1214 20:52:18.819053 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.819177 (+ 124us) service_pool.cc:225] Handling call
1214 20:52:20.366988 (+1547811us) ts_tablet_manager.cc:1151] Deleting tablet 2842f1dc0c374d24946f90dac1cf8aa5
1214 20:52:20.366996 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.380943 (+ 13947us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.384084 (+ 3141us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.474621 (+ 90537us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1535645}
I20251214 20:52:20.478225 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 49) took 1656 ms. Trace:
I20251214 20:52:20.469470 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:20.478437 1856 rpcz_store.cc:276] 1214 20:52:18.821887 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.821963 (+ 76us) service_pool.cc:225] Handling call
1214 20:52:20.478067 (+1656104us) ts_tablet_manager.cc:1151] Deleting tablet 06381efe75284e649fd6d1a9903cfa9b
1214 20:52:20.478076 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.478188 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646131}
I20251214 20:52:20.479513 1856 tablet_replica.cc:333] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.479722 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 2842f1dc0c374d24946f90dac1cf8aa5 (table default.Table_a2a64eab2d9a493da6fee1051eadd37d [id=d929401b9c4c4b4ba21353fb8a8ecc3e]) successfully deleted
I20251214 20:52:20.479985 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 88b9ca42a5ce4572aebe2a24f4758b12 (table default.Table_67fa945ac69a4a599f3dd43cfa93c0ff [id=0d173bfba2b343dbac6a02a5ff84c828]). Expected version 1 got 0
I20251214 20:52:20.483155 1202 tablet_bootstrap.cc:492] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:20.483289 1856 raft_consensus.cc:2243] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.483583 1856 raft_consensus.cc:2272] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.484164 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 8e11723c042a4637839f1b13bed0935a (DEFAULT_TABLE table=default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.485294 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e11723c042a4637839f1b13bed0935a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:20.480172 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 06381efe75284e649fd6d1a9903cfa9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06381efe75284e649fd6d1a9903cfa9b already in progress: creating tablet
W20251214 20:52:20.485498 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 292a28b12df246a8bc855d81c530746e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292a28b12df246a8bc855d81c530746e already in progress: creating tablet
I20251214 20:52:20.488765 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]). Expected version 1 got 0
I20251214 20:52:20.488958 1203 tablet_bootstrap.cc:492] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:20.489248 1203 ts_tablet_manager.cc:1403] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.002s
I20251214 20:52:20.489435 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 1 got 0
I20251214 20:52:20.490244 1203 raft_consensus.cc:359] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.490581 1203 raft_consensus.cc:385] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.490691 1203 raft_consensus.cc:740] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:20.490902 713 consensus_peers.cc:597] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3a2143b5508445492b970d2588cf1cf. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:20.491863 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2688515bd1441fbad7516f8343ca775 has no consensus state
I20251214 20:52:20.492833 1206 tablet_bootstrap.cc:492] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:20.493413 1206 ts_tablet_manager.cc:1403] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.283s user 0.004s sys 0.000s
I20251214 20:52:20.493788 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
W20251214 20:52:20.494024 713 consensus_peers.cc:597] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:20.494647 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1]). Expected version 1 got 0
W20251214 20:52:20.494913 713 consensus_peers.cc:597] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c25832da5c1644329e18653000f3330e. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:20.495101 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:20.495467 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 0d7c7b3f62764ff497f4d70b53eed7de (table default.Table_e26cdfc4ab8f4f19b52de34448bd70e5 [id=756e7314dcb5446f8e62baab11878de2]). Expected version 1 got 0
I20251214 20:52:20.495390 1206 raft_consensus.cc:359] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.495606 1206 raft_consensus.cc:385] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.495695 1206 raft_consensus.cc:740] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.495961 1206 consensus_queue.cc:260] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.496539 1206 ts_tablet_manager.cc:1434] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:20.496891 1206 tablet_bootstrap.cc:492] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:20.498060 750 tablet_service.cc:1505] Processing CreateTablet for tablet 8e11723c042a4637839f1b13bed0935a (DEFAULT_TABLE table=default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.498531 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e11723c042a4637839f1b13bed0935a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.498875 1206 tablet_bootstrap.cc:654] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.499728 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_55fe32e16f364e58afdd57c80a8966ae [id=6157817e96d642d09f9941ad7f7e82ee]). Expected version 1 got 0
I20251214 20:52:20.499939 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:20.500526 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:20.500743 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 3fa0317e999b483788e5c5652e6e83de (table default.Table_6c8f53a62dca43e2b8189d4a1fda9989 [id=ef22180b36a241fea13d0e622c02b50a]). Expected version 2 got 1
I20251214 20:52:20.501328 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
W20251214 20:52:20.501556 713 consensus_peers.cc:597] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0d661a1bb034053a4634ff537eafc7d. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:20.501681 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:20.508370 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:20.509138 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_b849df93c2394733b20258632b62e6b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.510313 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b849df93c2394733b20258632b62e6b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.511750 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:20.512061 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 0c973e0111804b8b869068bdcc5869bf (table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f]). Expected version 1 got 0
I20251214 20:52:20.512727 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet e16fe751f10f4616a00d0af811998729 (table default.Table_7923a9cba4ce4b27a12617e394ffbf2e [id=8fd9be1206a74aa1921e0dade8be3774]). Expected version 1 got 0
I20251214 20:52:20.512972 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:20.513334 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:20.513576 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:20.513886 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 02e256d4537249a394b80dc36beefef4 (table default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]). Expected version 1 got 0
I20251214 20:52:20.514154 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet cf04f91e79754569a51d468ab576b0dc (table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d]). Expected version 1 got 0
I20251214 20:52:20.514300 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 1 got 0
I20251214 20:52:20.514846 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d01db63fd5d2474a9defdce0b98fc29d (table default.Table_13f016135de44cf78187fbc4f9f8c622 [id=fc160acc971e4f7d8290560ee8aab065]). Expected version 1 got 0
I20251214 20:52:20.484283 1856 ts_tablet_manager.cc:1916] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:20.496366 989 leader_election.cc:343] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.515496 989 leader_election.cc:304] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:20.515676 1203 consensus_queue.cc:260] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.485757 1977 tablet_replica.cc:333] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
W20251214 20:52:20.516177 989 consensus_peers.cc:597] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe4086fc7a4f4db0b7541f06c9833aab. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:20.516238 1202 ts_tablet_manager.cc:1403] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.199s user 0.002s sys 0.003s
I20251214 20:52:20.516315 1977 raft_consensus.cc:2243] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.516578 1977 raft_consensus.cc:2272] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.516603 1882 raft_consensus.cc:2749] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.517220 1202 raft_consensus.cc:359] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.517414 1202 raft_consensus.cc:385] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.517511 1202 raft_consensus.cc:740] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.518760 1203 ts_tablet_manager.cc:1434] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20251214 20:52:20.488536 892 tablet_service.cc:1505] Processing CreateTablet for tablet 8e11723c042a4637839f1b13bed0935a (DEFAULT_TABLE table=default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:20.519639 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e11723c042a4637839f1b13bed0935a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:20.522931 1203 tablet_bootstrap.cc:492] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:20.524276 1977 ts_tablet_manager.cc:1916] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.524372 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b92e50faaffb45c0a3e1c7a94166f985" } new_table_name: "default.Table_5e0ee6bb376f49bb925ed69ce3a82727" modify_external_catalogs: true
I20251214 20:52:20.532820 1202 consensus_queue.cc:260] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.535936 1202 ts_tablet_manager.cc:1434] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20251214 20:52:20.540103 1203 tablet_bootstrap.cc:654] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.537814 1977 ts_tablet_manager.cc:1929] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251214 20:52:20.544091 1977 log.cc:1199] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/2b48bb260e854beb9ec61ce9c79b46c2
I20251214 20:52:20.544967 1977 ts_tablet_manager.cc:1950] T 2b48bb260e854beb9ec61ce9c79b46c2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.546372 1202 tablet_bootstrap.cc:492] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:20.549283 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]) successfully deleted
W20251214 20:52:20.551177 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 242a77ae57174de48f4eaf43384ee0ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 242a77ae57174de48f4eaf43384ee0ba already in progress: creating tablet
W20251214 20:52:20.552192 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 65e79db74fa046698507c83416ee1669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e79db74fa046698507c83416ee1669 already in progress: creating tablet
I20251214 20:52:20.553544 1977 tablet_replica.cc:333] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.556072 1977 raft_consensus.cc:2243] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.557861 1977 raft_consensus.cc:2272] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.560153 1202 tablet_bootstrap.cc:654] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.566973 1203 tablet_bootstrap.cc:492] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:20.567279 1203 ts_tablet_manager.cc:1403] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.000s
I20251214 20:52:20.568210 1203 raft_consensus.cc:359] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.569204 1206 tablet_bootstrap.cc:492] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:20.569391 1206 ts_tablet_manager.cc:1403] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.000s
I20251214 20:52:20.570416 1206 raft_consensus.cc:359] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.570787 1206 raft_consensus.cc:385] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.570892 1206 raft_consensus.cc:740] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.571820 1206 consensus_queue.cc:260] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.572660 1206 ts_tablet_manager.cc:1434] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251214 20:52:20.572835 1203 raft_consensus.cc:385] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.573211 1206 tablet_bootstrap.cc:492] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:20.573377 1203 raft_consensus.cc:740] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.573738 1203 consensus_queue.cc:260] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.576934 1206 tablet_bootstrap.cc:654] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.582410 1203 ts_tablet_manager.cc:1434] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251214 20:52:20.588846 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:20.562932 1977 ts_tablet_manager.cc:1916] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.595023 1856 ts_tablet_manager.cc:1929] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.595232 1203 tablet_bootstrap.cc:492] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:20.597630 1203 tablet_bootstrap.cc:654] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.598815 1963 raft_consensus.cc:493] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.599030 1963 raft_consensus.cc:515] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.599762 1963 leader_election.cc:290] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.600517 1202 tablet_bootstrap.cc:492] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:20.600805 1202 ts_tablet_manager.cc:1403] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.004s
W20251214 20:52:20.601219 713 leader_election.cc:343] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 52daf1204bf142e7b82852bdec6582fb
I20251214 20:52:20.601687 1982 ts_tablet_manager.cc:1929] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:20.601897 1982 log.cc:1199] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/8af99e0bd8774a678575847a9ec2e557
I20251214 20:52:20.602370 1982 ts_tablet_manager.cc:1950] T 8af99e0bd8774a678575847a9ec2e557 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:20.602636 1202 raft_consensus.cc:359] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.603788 1206 tablet_bootstrap.cc:492] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:20.604017 1206 ts_tablet_manager.cc:1403] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20251214 20:52:20.603968 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 8af99e0bd8774a678575847a9ec2e557 (table default.Table_ade67d7e10e04c6fb0cfc134947a559d [id=7986a565ce914875bf3e24c1cc2c288a]) successfully deleted
I20251214 20:52:20.604908 1982 tablet_replica.cc:333] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:20.604581 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52daf1204bf142e7b82852bdec6582fb" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:20.605178 1202 raft_consensus.cc:385] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.605357 1202 raft_consensus.cc:740] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.605672 1202 consensus_queue.cc:260] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.606097 1982 raft_consensus.cc:2243] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.605768 712 leader_election.cc:304] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:20.606668 1963 raft_consensus.cc:2749] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.604879 1206 raft_consensus.cc:359] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.607012 1206 raft_consensus.cc:385] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.607120 1206 raft_consensus.cc:740] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.607477 1206 consensus_queue.cc:260] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.611770 1206 ts_tablet_manager.cc:1434] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251214 20:52:20.613379 1206 tablet_bootstrap.cc:492] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:20.615298 1928 raft_consensus.cc:493] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.615564 1206 tablet_bootstrap.cc:654] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.615516 1928 raft_consensus.cc:515] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.616169 1928 leader_election.cc:290] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.606213 1856 log.cc:1199] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/8d74d3abc45a41969289c0a0892bfa04
I20251214 20:52:20.616856 1856 ts_tablet_manager.cc:1950] T 8d74d3abc45a41969289c0a0892bfa04 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.617092 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9b7149a436a4cab80b39adccf8567f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:20.617339 910 raft_consensus.cc:2468] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:20.617542 1982 raft_consensus.cc:2272] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.618060 991 leader_election.cc:304] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:20.618100 1982 ts_tablet_manager.cc:1916] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.618453 1928 raft_consensus.cc:2804] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:20.618539 1928 raft_consensus.cc:493] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:20.618628 1928 raft_consensus.cc:3060] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.619985 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 52) took 1786 ms. Trace:
I20251214 20:52:20.620090 1856 rpcz_store.cc:276] 1214 20:52:18.833261 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.833351 (+ 90us) service_pool.cc:225] Handling call
1214 20:52:20.479437 (+1646086us) ts_tablet_manager.cc:1151] Deleting tablet 8d74d3abc45a41969289c0a0892bfa04
1214 20:52:20.479448 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.588880 (+109432us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.594885 (+ 6005us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.619907 (+ 25022us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1639139}
I20251214 20:52:20.620602 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 8d74d3abc45a41969289c0a0892bfa04 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]) successfully deleted
I20251214 20:52:20.620700 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 51) took 1790 ms. Trace:
I20251214 20:52:20.620790 1856 rpcz_store.cc:276] 1214 20:52:18.831789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.832274 (+ 485us) service_pool.cc:225] Handling call
1214 20:52:20.620532 (+1788258us) ts_tablet_manager.cc:1151] Deleting tablet b1f4ea72c6274569a98cab6afdc55d55
1214 20:52:20.620542 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.620655 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1777434}
I20251214 20:52:20.621258 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 53) took 1784 ms. Trace:
I20251214 20:52:20.621348 1856 rpcz_store.cc:276] 1214 20:52:18.836409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.836463 (+ 54us) service_pool.cc:225] Handling call
1214 20:52:20.621105 (+1784642us) ts_tablet_manager.cc:1151] Deleting tablet 69f6108792fa4c2a9744e3248c76251f
1214 20:52:20.621113 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.621209 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1774467}
W20251214 20:52:20.621831 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b1f4ea72c6274569a98cab6afdc55d55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1f4ea72c6274569a98cab6afdc55d55 already in progress: creating tablet
W20251214 20:52:20.622088 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 69f6108792fa4c2a9744e3248c76251f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f6108792fa4c2a9744e3248c76251f already in progress: creating tablet
I20251214 20:52:20.623301 1856 tablet_replica.cc:333] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.623512 1856 raft_consensus.cc:2243] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.623673 1856 raft_consensus.cc:2272] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.623651 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9b7149a436a4cab80b39adccf8567f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.624151 1856 ts_tablet_manager.cc:1916] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:20.624425 989 leader_election.cc:343] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.589196 448 catalog_manager.cc:5654] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:20.625041 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_55fe32e16f364e58afdd57c80a8966ae [id=6157817e96d642d09f9941ad7f7e82ee]). Expected version 1 got 0
I20251214 20:52:20.625349 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 0d7c7b3f62764ff497f4d70b53eed7de (table default.Table_e26cdfc4ab8f4f19b52de34448bd70e5 [id=756e7314dcb5446f8e62baab11878de2]). Expected version 1 got 0
I20251214 20:52:20.625634 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:20.626019 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1]). Expected version 1 got 0
I20251214 20:52:20.628429 1928 raft_consensus.cc:515] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.629040 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]). Expected version 1 got 0
I20251214 20:52:20.629654 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9b7149a436a4cab80b39adccf8567f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:20.629539 448 catalog_manager.cc:5654] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:20.629930 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 88b9ca42a5ce4572aebe2a24f4758b12 (table default.Table_67fa945ac69a4a599f3dd43cfa93c0ff [id=0d173bfba2b343dbac6a02a5ff84c828]). Expected version 1 got 0
W20251214 20:52:20.630366 989 leader_election.cc:343] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.630723 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d01db63fd5d2474a9defdce0b98fc29d (table default.Table_13f016135de44cf78187fbc4f9f8c622 [id=fc160acc971e4f7d8290560ee8aab065]). Expected version 1 got 0
I20251214 20:52:20.630976 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:20.633123 1982 ts_tablet_manager.cc:1929] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.633252 1982 log.cc:1199] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/9523d72fd91445539d67e1eac2a80afd
I20251214 20:52:20.633459 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:20.633504 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9b7149a436a4cab80b39adccf8567f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:20.633661 1206 tablet_bootstrap.cc:492] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:20.633733 1982 ts_tablet_manager.cc:1950] T 9523d72fd91445539d67e1eac2a80afd P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:20.633745 910 raft_consensus.cc:3060] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.633788 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:20.633838 1206 ts_tablet_manager.cc:1403] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251214 20:52:20.633965 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_d7519f28730149dc956626fb163db031 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 1 got 0
I20251214 20:52:20.634703 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3fa0317e999b483788e5c5652e6e83de (table default.Table_6c8f53a62dca43e2b8189d4a1fda9989 [id=ef22180b36a241fea13d0e622c02b50a]). Expected version 2 got 1
I20251214 20:52:20.634730 1206 raft_consensus.cc:359] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.634922 1206 raft_consensus.cc:385] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.635007 1206 raft_consensus.cc:740] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.635268 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 1 got 0
I20251214 20:52:20.635283 1206 consensus_queue.cc:260] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.635628 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:20.636349 1206 ts_tablet_manager.cc:1434] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251214 20:52:20.635924 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 9523d72fd91445539d67e1eac2a80afd (table default.Table_76a2939ccdea4fdeaa1da24713bc3ca7 [id=edb7e58d7c0b4ddea05ba0139f321042]) successfully deleted
W20251214 20:52:20.637233 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2a7fc15882e2474bb6885070be13b1fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a7fc15882e2474bb6885070be13b1fa already in progress: creating tablet
W20251214 20:52:20.637439 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 7fcfcfe81fe24d369eea696efc61d21c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fcfcfe81fe24d369eea696efc61d21c already in progress: creating tablet
W20251214 20:52:20.637606 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3e5e220d083243eb8aa4328f366cd7ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e5e220d083243eb8aa4328f366cd7ff already in progress: creating tablet
I20251214 20:52:20.638103 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:20.638311 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e16fe751f10f4616a00d0af811998729 (table default.Table_7923a9cba4ce4b27a12617e394ffbf2e [id=8fd9be1206a74aa1921e0dade8be3774]). Expected version 1 got 0
I20251214 20:52:20.638432 910 raft_consensus.cc:2468] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:20.635270 1982 tablet_replica.cc:333] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:20.638959 1982 raft_consensus.cc:2243] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.639058 991 leader_election.cc:304] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:20.639211 1982 raft_consensus.cc:2272] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.639667 1915 raft_consensus.cc:2804] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:20.639820 1915 raft_consensus.cc:697] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:20.639822 1982 ts_tablet_manager.cc:1916] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.640791 1915 consensus_queue.cc:237] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.642316 1856 ts_tablet_manager.cc:1929] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:20.642742 1856 log.cc:1199] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/324379e3217442c3addcfe58060b9ad7
I20251214 20:52:20.643368 1856 ts_tablet_manager.cc:1950] T 324379e3217442c3addcfe58060b9ad7 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.643646 1928 leader_election.cc:290] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.644228 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 54) took 1800 ms. Trace:
I20251214 20:52:20.645382 1206 tablet_bootstrap.cc:492] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:20.645529 1856 rpcz_store.cc:276] 1214 20:52:18.843534 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.844285 (+ 751us) service_pool.cc:225] Handling call
1214 20:52:20.623222 (+1778937us) ts_tablet_manager.cc:1151] Deleting tablet 324379e3217442c3addcfe58060b9ad7
1214 20:52:20.623231 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.630732 (+ 7501us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.642151 (+ 11419us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.644183 (+ 2032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1774692}
I20251214 20:52:20.646528 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 324379e3217442c3addcfe58060b9ad7 (table default.Table_ff05c589ffd8444daa99c956e92c37e6 [id=16a8869e1ba641438d0836f6f204414f]) successfully deleted
I20251214 20:52:20.647260 1206 tablet_bootstrap.cc:654] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.646875 1856 tablet_replica.cc:333] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.649441 1856 raft_consensus.cc:2243] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.638729 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:20.649884 1856 raft_consensus.cc:2272] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.650188 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:20.650548 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:20.650889 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:20.650995 1856 ts_tablet_manager.cc:1916] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.651250 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:20.653931 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:20.654165 1202 ts_tablet_manager.cc:1434] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.053s user 0.001s sys 0.002s
I20251214 20:52:20.654659 1928 raft_consensus.cc:493] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.654964 1928 raft_consensus.cc:515] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.656214 1928 leader_election.cc:290] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.656841 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d72f55d2d2440d993aa205d96c356f1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:20.657697 1202 tablet_bootstrap.cc:492] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:20.654229 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 02e256d4537249a394b80dc36beefef4 (table default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]). Expected version 1 got 0
I20251214 20:52:20.659986 1203 tablet_bootstrap.cc:492] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:20.660225 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 0c973e0111804b8b869068bdcc5869bf (table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f]). Expected version 1 got 0
I20251214 20:52:20.657958 910 raft_consensus.cc:2468] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:20.661753 448 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 727e7b61ae3942098712720d344c5310 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]). Expected version 1 got 0
I20251214 20:52:20.662278 452 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974] alter complete (version 1)
I20251214 20:52:20.663044 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 762068d560ff47918dd9fd8791083433
I20251214 20:52:20.663197 1202 tablet_bootstrap.cc:654] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.663951 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_9321256005b04155b2f5a451ce5065fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.664593 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9321256005b04155b2f5a451ce5065fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.665934 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 762068d560ff47918dd9fd8791083433 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:20.667038 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet feb19ea389af4ea6bb158a0e8aaeb094 consensus state has no leader
W20251214 20:52:20.667047 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 738668adf2874d6894db6fa875f6990c consensus state has no leader
I20251214 20:52:20.667474 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_1e4dafc4ff8940ea90956623c915e3fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:20.660372 1882 raft_consensus.cc:493] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.667822 1882 raft_consensus.cc:515] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:20.668012 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e4dafc4ff8940ea90956623c915e3fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.668565 1882 leader_election.cc:290] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.668668 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 2accf8a160584c82bb695a52e3f2b05c
I20251214 20:52:20.661087 991 leader_election.cc:304] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:20.669404 1882 raft_consensus.cc:2804] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:20.669385 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_4ad0cc66006940d2abacd388e227075a" } modify_external_catalogs: true
I20251214 20:52:20.668033 450 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10] alter complete (version 3)
I20251214 20:52:20.669504 1882 raft_consensus.cc:493] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:20.669574 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_4ad0cc66006940d2abacd388e227075a" } modify_external_catalogs: true
I20251214 20:52:20.669620 1882 raft_consensus.cc:3060] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.672869 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 2accf8a160584c82bb695a52e3f2b05c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.672844 1882 raft_consensus.cc:515] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:20.673578 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1389716f5ce44f588c21a34c7a7d4c8 consensus state has no leader
I20251214 20:52:20.673933 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3736aabecd40417f9cb502ac612bacee" } new_table_name: "default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3" modify_external_catalogs: true
I20251214 20:52:20.674026 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 2accf8a160584c82bb695a52e3f2b05c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.674671 1882 leader_election.cc:290] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.674897 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d72f55d2d2440d993aa205d96c356f1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:20.675112 904 raft_consensus.cc:3060] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.677106 1203 ts_tablet_manager.cc:1403] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.088s user 0.000s sys 0.004s
I20251214 20:52:20.678234 1203 raft_consensus.cc:359] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:20.678676 904 raft_consensus.cc:2468] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:20.667529 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_14f3cdeac59a4c20988d0e23744d2fbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.681007 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14f3cdeac59a4c20988d0e23744d2fbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.681069 1762 raft_consensus.cc:493] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.681247 1762 raft_consensus.cc:515] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.681903 1762 leader_election.cc:290] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:20.682049 1650 raft_consensus.cc:493] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.682199 1650 raft_consensus.cc:515] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.682868 1650 leader_election.cc:290] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:20.686355 1977 ts_tablet_manager.cc:1929] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:20.664155 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 762068d560ff47918dd9fd8791083433 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.679203 991 leader_election.cc:304] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:20.687312 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a7fc15882e2474bb6885070be13b1fa" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:20.687590 1915 raft_consensus.cc:2804] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:20.688004 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e256d4537249a394b80dc36beefef4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:20.688191 1915 raft_consensus.cc:697] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:20.688575 1915 consensus_queue.cc:237] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.689284 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_873207e85b954e21b3d98ad4c02a0248"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.689828 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_873207e85b954e21b3d98ad4c02a0248 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.691883 1203 raft_consensus.cc:385] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.692028 1203 raft_consensus.cc:740] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.692135 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d528fb3647e34ed5b6b9199d7c6db3a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.692857 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d528fb3647e34ed5b6b9199d7c6db3a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.693163 1203 consensus_queue.cc:260] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:20.687945 851 leader_election.cc:343] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:20.693728 851 leader_election.cc:343] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.694233 1977 log.cc:1199] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/25974e5db555447baf5799775d15cd14
I20251214 20:52:20.694674 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 762068d560ff47918dd9fd8791083433 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.694726 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d72f55d2d2440d993aa205d96c356f1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.694802 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a7fc15882e2474bb6885070be13b1fa" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.661667 1856 ts_tablet_manager.cc:1929] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251214 20:52:20.695348 848 leader_election.cc:343] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.695525 1856 log.cc:1199] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/c4b22043d81b493fb12116422b1d3fb1
I20251214 20:52:20.695498 848 leader_election.cc:304] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:20.695647 1977 ts_tablet_manager.cc:1950] T 25974e5db555447baf5799775d15cd14 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:20.695928 1650 raft_consensus.cc:2749] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.696097 1928 raft_consensus.cc:493] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.696134 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf37c50e511f454ca4887c7f6ca1a746" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:20.696223 1928 raft_consensus.cc:515] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:20.695058 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 2accf8a160584c82bb695a52e3f2b05c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.696650 910 raft_consensus.cc:2468] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:20.696880 1928 leader_election.cc:290] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:20.697911 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9f198ef4dfa4f5791144b9a24fe693f consensus state has no leader
I20251214 20:52:20.698081 991 leader_election.cc:304] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:20.698352 1856 ts_tablet_manager.cc:1950] T c4b22043d81b493fb12116422b1d3fb1 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.698578 1667 raft_consensus.cc:2804] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:20.698675 1667 raft_consensus.cc:493] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:20.698781 1667 raft_consensus.cc:3060] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.700022 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 62) took 1705 ms. Trace:
I20251214 20:52:20.702862 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf37c50e511f454ca4887c7f6ca1a746" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.703264 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e256d4537249a394b80dc36beefef4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:20.703764 848 leader_election.cc:343] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.703653 1856 rpcz_store.cc:276] 1214 20:52:18.994555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.994675 (+ 120us) service_pool.cc:225] Handling call
1214 20:52:20.646800 (+1652125us) ts_tablet_manager.cc:1151] Deleting tablet c4b22043d81b493fb12116422b1d3fb1
1214 20:52:20.646811 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.655896 (+ 9085us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.661548 (+ 5652us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.699951 (+ 38403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651716}
I20251214 20:52:20.703922 848 leader_election.cc:304] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:20.704144 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet c4b22043d81b493fb12116422b1d3fb1 (table default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]) successfully deleted
I20251214 20:52:20.704301 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 63) took 1709 ms. Trace:
I20251214 20:52:20.704387 1650 raft_consensus.cc:2749] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.704383 1856 rpcz_store.cc:276] 1214 20:52:18.994687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.995176 (+ 489us) service_pool.cc:225] Handling call
1214 20:52:20.704132 (+1708956us) ts_tablet_manager.cc:1151] Deleting tablet a063083f54ad49d2a842fa080d44ed85
1214 20:52:20.704144 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.704255 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1708690}
W20251214 20:52:20.704730 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a063083f54ad49d2a842fa080d44ed85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a063083f54ad49d2a842fa080d44ed85 already in progress: creating tablet
I20251214 20:52:20.704852 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 71) took 1708 ms. Trace:
I20251214 20:52:20.704923 1856 rpcz_store.cc:276] 1214 20:52:18.995870 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.995939 (+ 69us) service_pool.cc:225] Handling call
1214 20:52:20.704699 (+1708760us) ts_tablet_manager.cc:1151] Deleting tablet 3d949abef1cb41a9905482d81a98d2c1
1214 20:52:20.704718 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.704805 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1708536}
I20251214 20:52:20.705312 1856 tablet_replica.cc:333] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:20.705276 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3d949abef1cb41a9905482d81a98d2c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d949abef1cb41a9905482d81a98d2c1 already in progress: creating tablet
I20251214 20:52:20.705504 1856 raft_consensus.cc:2243] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.705684 1856 raft_consensus.cc:2272] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.706070 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bbb28259c1442ae94f4eff76c25c539" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.706233 1856 ts_tablet_manager.cc:1916] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:20.706781 989 leader_election.cc:343] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:20.707175 989 leader_election.cc:343] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.707041 450 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_0870823e1f29409f9d61ebee0bbd060d [id=0a848dd5f6474712b2eba70add3743dc] alter complete (version 3)
I20251214 20:52:20.708807 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d72f55d2d2440d993aa205d96c356f1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:20.711658 1982 ts_tablet_manager.cc:1929] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:20.712347 1982 log.cc:1199] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/52daf1204bf142e7b82852bdec6582fb
I20251214 20:52:20.702183 1667 raft_consensus.cc:515] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.713219 1982 ts_tablet_manager.cc:1950] T 52daf1204bf142e7b82852bdec6582fb P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:20.713416 1667 leader_election.cc:290] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.714351 1982 tablet_replica.cc:333] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:20.714560 1982 raft_consensus.cc:2243] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.714532 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf37c50e511f454ca4887c7f6ca1a746" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:20.714726 1982 raft_consensus.cc:2272] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.714797 910 raft_consensus.cc:3060] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.715272 1982 ts_tablet_manager.cc:1916] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.717989 910 raft_consensus.cc:2468] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:20.718755 991 leader_election.cc:304] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:20.719892 1928 raft_consensus.cc:2804] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:20.720026 1856 ts_tablet_manager.cc:1929] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.720669 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 52daf1204bf142e7b82852bdec6582fb (table default.Table_5fbbdeeea9a14e17bf6f9d0cc4a56d98 [id=a0e13ae34009499d91b4461520c292c1]) successfully deleted
W20251214 20:52:20.721259 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet d6c0f445cd7e44769d79ad5566fd8d97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6c0f445cd7e44769d79ad5566fd8d97 already in progress: creating tablet
I20251214 20:52:20.721882 1982 ts_tablet_manager.cc:1929] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.720120 1928 raft_consensus.cc:697] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:20.725302 1928 consensus_queue.cc:237] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:20.722975 991 leader_election.cc:343] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 3bbb28259c1442ae94f4eff76c25c539
I20251214 20:52:20.727065 1856 log.cc:1199] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/6bd6856b800247de896f16eee66f2827
I20251214 20:52:20.727814 1856 ts_tablet_manager.cc:1950] T 6bd6856b800247de896f16eee66f2827 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.728091 1982 log.cc:1199] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/472e4a5990e64d02b1d52dc6e86d7190
I20251214 20:52:20.729557 1982 ts_tablet_manager.cc:1950] T 472e4a5990e64d02b1d52dc6e86d7190 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:20.729127 1206 tablet_bootstrap.cc:492] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:20.730053 1206 ts_tablet_manager.cc:1403] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.085s user 0.009s sys 0.005s
I20251214 20:52:20.730870 1206 raft_consensus.cc:359] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.731182 1206 raft_consensus.cc:385] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.731303 1206 raft_consensus.cc:740] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.729717 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 56) took 1878 ms. Trace:
I20251214 20:52:20.731647 1206 consensus_queue.cc:260] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.731597 1856 rpcz_store.cc:276] 1214 20:52:18.851679 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.852422 (+ 743us) service_pool.cc:225] Handling call
1214 20:52:20.705238 (+1852816us) ts_tablet_manager.cc:1151] Deleting tablet 6bd6856b800247de896f16eee66f2827
1214 20:52:20.705247 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.710988 (+ 5741us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.719876 (+ 8888us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.729661 (+ 9785us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1704130}
I20251214 20:52:20.732069 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 6bd6856b800247de896f16eee66f2827 (table default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]) successfully deleted
I20251214 20:52:20.732340 1203 ts_tablet_manager.cc:1434] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.055s user 0.001s sys 0.001s
I20251214 20:52:20.732828 1206 ts_tablet_manager.cc:1434] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251214 20:52:20.733263 1206 tablet_bootstrap.cc:492] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:20.735074 1206 tablet_bootstrap.cc:654] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.736275 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 472e4a5990e64d02b1d52dc6e86d7190 (table default.Table_2acb51a56a60469e941100c99494a99b [id=b16533ff3115482d9357bd88bc1406cd]) successfully deleted
I20251214 20:52:20.732909 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 57) took 1881 ms. Trace:
I20251214 20:52:20.738372 1856 rpcz_store.cc:276] 1214 20:52:18.851786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.852451 (+ 665us) service_pool.cc:225] Handling call
1214 20:52:20.732342 (+1879891us) ts_tablet_manager.cc:1151] Deleting tablet bd8226b431c6456885e928880938969d
1214 20:52:20.732354 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.732875 (+ 521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1730910}
I20251214 20:52:20.736053 1982 tablet_replica.cc:333] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:20.739831 1982 raft_consensus.cc:2243] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.740082 1982 raft_consensus.cc:2272] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.740509 1856 tablet_replica.cc:333] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:20.740511 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet bd8226b431c6456885e928880938969d because the tablet was not found. No further retry: Not found: Tablet not found: bd8226b431c6456885e928880938969d
I20251214 20:52:20.740797 1856 raft_consensus.cc:2243] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.741039 1856 raft_consensus.cc:2272] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.741724 1856 ts_tablet_manager.cc:1916] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:20.741894 989 consensus_peers.cc:597] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:20.742622 1982 ts_tablet_manager.cc:1916] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.742640 1928 raft_consensus.cc:493] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.742784 1928 raft_consensus.cc:515] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.743952 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_08a9cb7994804ae586d3a7e3de4adb7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.744596 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08a9cb7994804ae586d3a7e3de4adb7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.744779 1928 leader_election.cc:290] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:20.745447 991 leader_election.cc:343] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 604b7c6fe3b24daca70711a1990af556
W20251214 20:52:20.742784 989 leader_election.cc:343] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.745740 989 leader_election.cc:304] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:20.746203 989 leader_election.cc:343] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.746574 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_eb4f37e15d9a4b1b9e3892c7891329bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.747366 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb4f37e15d9a4b1b9e3892c7891329bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.747632 1928 raft_consensus.cc:2749] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.748232 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf37c50e511f454ca4887c7f6ca1a746" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:20.752112 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "604b7c6fe3b24daca70711a1990af556" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.752143 1856 ts_tablet_manager.cc:1929] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.752305 1856 log.cc:1199] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/d768c45bc02b4d108798efbf3f6aaf7d
I20251214 20:52:20.756063 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]) successfully deleted
W20251214 20:52:20.756582 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 762068d560ff47918dd9fd8791083433 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 762068d560ff47918dd9fd8791083433 already in progress: creating tablet
I20251214 20:52:20.756678 1856 ts_tablet_manager.cc:1950] T d768c45bc02b4d108798efbf3f6aaf7d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:20.756847 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2accf8a160584c82bb695a52e3f2b05c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2accf8a160584c82bb695a52e3f2b05c already in progress: creating tablet
W20251214 20:52:20.754592 989 leader_election.cc:343] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.738901 1203 tablet_bootstrap.cc:492] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:20.758710 1202 tablet_bootstrap.cc:492] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:20.759966 1203 tablet_bootstrap.cc:654] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.760004 1982 ts_tablet_manager.cc:1929] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251214 20:52:20.760666 989 leader_election.cc:343] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.762430 989 leader_election.cc:304] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:20.762758 1882 raft_consensus.cc:2749] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.763118 1202 ts_tablet_manager.cc:1403] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.003s
I20251214 20:52:20.763181 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 58) took 1905 ms. Trace:
I20251214 20:52:20.763299 1856 rpcz_store.cc:276] 1214 20:52:18.851881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.851950 (+ 69us) service_pool.cc:225] Handling call
1214 20:52:20.740418 (+1888468us) ts_tablet_manager.cc:1151] Deleting tablet d768c45bc02b4d108798efbf3f6aaf7d
1214 20:52:20.740428 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.745797 (+ 5369us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.752014 (+ 6217us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.757497 (+ 5483us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1716382}
I20251214 20:52:20.764393 1202 raft_consensus.cc:359] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.764747 1202 raft_consensus.cc:385] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.764897 1202 raft_consensus.cc:740] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.765228 1202 consensus_queue.cc:260] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.761623 1206 tablet_bootstrap.cc:492] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:20.766005 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet d768c45bc02b4d108798efbf3f6aaf7d (table default.Table_ca875b1006554ba6845c0cf23671a8da [id=8c192689a28f4314bb5608c08ea21a2e]) successfully deleted
I20251214 20:52:20.765743 1206 ts_tablet_manager.cc:1403] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.001s
I20251214 20:52:20.767017 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_9e2910a477ea4326b888fa02a5fa5a79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:20.767360 1856 tablet_replica.cc:333] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.767323 1206 raft_consensus.cc:359] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.767513 1206 raft_consensus.cc:385] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.767590 1206 raft_consensus.cc:740] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
W20251214 20:52:20.767764 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e2910a477ea4326b888fa02a5fa5a79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.767797 1206 consensus_queue.cc:260] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.768271 1206 ts_tablet_manager.cc:1434] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:20.768581 1206 tablet_bootstrap.cc:492] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:20.768756 1856 raft_consensus.cc:2243] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.768955 1856 raft_consensus.cc:2272] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.769538 1856 ts_tablet_manager.cc:1916] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.770814 1206 tablet_bootstrap.cc:654] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.766000 1962 raft_consensus.cc:493] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.771414 1962 raft_consensus.cc:515] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.772086 1962 leader_election.cc:290] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.773577 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 0d7c7b3f62764ff497f4d70b53eed7de
I20251214 20:52:20.774235 447 master_service.cc:954] ReplaceTablet: received request to replace tablet a0279992d1f2419ab1caae116a0391b2 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:20.774771 447 master_service.cc:954] ReplaceTablet: received request to replace tablet 16816fa6b8c34207bb929147910b93cb from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:20.778041 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 1dca369dc5f54a4ea1783e334c5670d6
I20251214 20:52:20.778307 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d7c7b3f62764ff497f4d70b53eed7de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.778789 454 master_service.cc:954] ReplaceTablet: received request to replace tablet 0d7c7b3f62764ff497f4d70b53eed7de from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:20.779063 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d7c7b3f62764ff497f4d70b53eed7de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.779146 450 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151] alter complete (version 1)
I20251214 20:52:20.779446 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dca369dc5f54a4ea1783e334c5670d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.779551 1982 log.cc:1199] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/b652a8a80246443491551e76e68d2a8b
I20251214 20:52:20.779948 1982 ts_tablet_manager.cc:1950] T b652a8a80246443491551e76e68d2a8b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:20.780256 1203 tablet_bootstrap.cc:492] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:20.780400 1203 ts_tablet_manager.cc:1403] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20251214 20:52:20.780773 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_91b2da75e7104e0f95ed470c749f843f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:20.781059 1982 tablet_replica.cc:333] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:20.781239 1982 raft_consensus.cc:2243] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.781184 1203 raft_consensus.cc:359] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.781353 1203 raft_consensus.cc:385] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251214 20:52:20.781399 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91b2da75e7104e0f95ed470c749f843f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.781431 1203 raft_consensus.cc:740] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.781585 1982 raft_consensus.cc:2272] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.781683 1203 consensus_queue.cc:260] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.782107 1982 ts_tablet_manager.cc:1916] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.782738 1203 ts_tablet_manager.cc:1434] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:20.782826 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ba612520b0be474db6573ce754044c8a" } modify_external_catalogs: true
I20251214 20:52:20.782982 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ba612520b0be474db6573ce754044c8a" } modify_external_catalogs: true
I20251214 20:52:20.783067 1203 tablet_bootstrap.cc:492] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:20.783618 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d7c7b3f62764ff497f4d70b53eed7de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.786518 1203 tablet_bootstrap.cc:654] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:20.787472 713 leader_election.cc:343] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: c2ae9dc8ead8403cb7215af8778872f0
I20251214 20:52:20.789438 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet b652a8a80246443491551e76e68d2a8b (table default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]) successfully deleted
W20251214 20:52:20.789745 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b315744c8cd043c5a8bdceec73a7e28b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b315744c8cd043c5a8bdceec73a7e28b already in progress: creating tablet
I20251214 20:52:20.789888 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dca369dc5f54a4ea1783e334c5670d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.790905 1856 ts_tablet_manager.cc:1929] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.791004 1856 log.cc:1199] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ba50e32f7e724868bb891045a939e362
I20251214 20:52:20.792326 1202 ts_tablet_manager.cc:1434] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
W20251214 20:52:20.796113 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 0d7c7b3f62764ff497f4d70b53eed7de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d7c7b3f62764ff497f4d70b53eed7de already in progress: creating tablet
I20251214 20:52:20.797822 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dca369dc5f54a4ea1783e334c5670d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.797971 1882 raft_consensus.cc:493] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.798102 1882 raft_consensus.cc:515] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:20.798300 451 master_service.cc:954] ReplaceTablet: received request to replace tablet c8e33ba1149a44c294885d76f676160a from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:20.798738 1882 leader_election.cc:290] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.799307 1856 ts_tablet_manager.cc:1950] T ba50e32f7e724868bb891045a939e362 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.800261 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 72) took 1804 ms. Trace:
I20251214 20:52:20.800400 1856 rpcz_store.cc:276] 1214 20:52:18.995992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.997254 (+ 1262us) service_pool.cc:225] Handling call
1214 20:52:20.767268 (+1770014us) ts_tablet_manager.cc:1151] Deleting tablet ba50e32f7e724868bb891045a939e362
1214 20:52:20.767281 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.777706 (+ 10425us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.790770 (+ 13064us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.800224 (+ 9454us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742871}
I20251214 20:52:20.801311 1206 tablet_bootstrap.cc:492] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
W20251214 20:52:20.801440 712 leader_election.cc:343] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: c2ae9dc8ead8403cb7215af8778872f0
I20251214 20:52:20.801542 1206 ts_tablet_manager.cc:1403] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.004s
I20251214 20:52:20.801537 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f4760585baf4b4ab3b9b94b5a748a61" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:20.801625 712 leader_election.cc:304] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:20.801779 912 raft_consensus.cc:2468] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:20.801928 1856 tablet_replica.cc:333] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.802126 1962 raft_consensus.cc:2749] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:20.802155 1856 raft_consensus.cc:2243] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.802481 1856 pending_rounds.cc:70] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:20.802435 1206 raft_consensus.cc:359] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.802551 1856 pending_rounds.cc:77] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493716388302848 op_type: NO_OP noop_request { }
I20251214 20:52:20.802654 1206 raft_consensus.cc:385] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.802649 1856 raft_consensus.cc:2889] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:20.802716 1206 raft_consensus.cc:740] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.802762 1856 raft_consensus.cc:2272] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.802934 1206 consensus_queue.cc:260] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.804267 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ba50e32f7e724868bb891045a939e362 (table default.Table_8e3f7e1435ea493fa4729ca0d9e23573 [id=c3e5006d02fc4b69987feb30ba66d069]) successfully deleted
I20251214 20:52:20.804486 991 leader_election.cc:304] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:20.806401 2022 raft_consensus.cc:2804] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:20.806636 2022 raft_consensus.cc:493] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:20.806888 2022 raft_consensus.cc:3060] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.809567 1206 ts_tablet_manager.cc:1434] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20251214 20:52:20.810604 2022 raft_consensus.cc:515] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:20.813405 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f4760585baf4b4ab3b9b94b5a748a61" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:20.813687 910 raft_consensus.cc:3060] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.813968 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f4760585baf4b4ab3b9b94b5a748a61" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:20.814395 1206 tablet_bootstrap.cc:492] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:20.815331 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f4760585baf4b4ab3b9b94b5a748a61" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:20.815960 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 16816fa6b8c34207bb929147910b93cb
W20251214 20:52:20.816002 989 leader_election.cc:343] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.816511 447 catalog_manager.cc:6724] ReplaceTablet: tablet 16816fa6b8c34207bb929147910b93cb deleted and replaced by tablet 532c44785f204f988518c6fd628f76ca
I20251214 20:52:20.816613 910 raft_consensus.cc:2468] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:20.816826 447 master_service.cc:954] ReplaceTablet: received request to replace tablet 3e5e220d083243eb8aa4328f366cd7ff from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:20.817075 991 leader_election.cc:304] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:20.817402 1882 raft_consensus.cc:2804] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:20.817415 447 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6c8f53a62dca43e2b8189d4a1fda9989" } new_table_name: "default.Table_3c676f861c28408f99faac2ab8f7eec1" modify_external_catalogs: true
I20251214 20:52:20.817572 1882 raft_consensus.cc:697] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
W20251214 20:52:20.817761 989 leader_election.cc:343] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:20.817894 1882 consensus_queue.cc:237] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:20.818048 448 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_31645b7a21d345ef9bc06c0ba9b2c5fa [id=8cd62da3d57c4be4b5f10ace49d2e148] alter complete (version 4)
I20251214 20:52:20.819092 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 16816fa6b8c34207bb929147910b93cb with delete_type TABLET_DATA_DELETED (replaced by tablet 532c44785f204f988518c6fd628f76ca) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.823477 2022 leader_election.cc:290] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.824810 450 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d] alter complete (version 1)
I20251214 20:52:20.826773 1206 tablet_bootstrap.cc:654] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.830825 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.830989 887 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.831184 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7c6fe3b24daca70711a1990af556 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.832136 1982 ts_tablet_manager.cc:1929] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.832208 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b48bb260e854beb9ec61ce9c79b46c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 566c6e4761d5486d964e276d151e32b0) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.832471 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 16816fa6b8c34207bb929147910b93cb with delete_type TABLET_DATA_DELETED (replaced by tablet 532c44785f204f988518c6fd628f76ca) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.835460 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 16816fa6b8c34207bb929147910b93cb with delete_type TABLET_DATA_DELETED (replaced by tablet 532c44785f204f988518c6fd628f76ca) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.835731 1982 log.cc:1199] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/d768c45bc02b4d108798efbf3f6aaf7d
I20251214 20:52:20.836227 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 9523d72fd91445539d67e1eac2a80afd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.836558 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.836725 894 tablet_service.cc:1552] Processing DeleteTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf with delete_type TABLET_DATA_DELETED (replaced by tablet 6fef1d6add764fd28a8f4995b337ea71) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.836798 895 tablet_service.cc:1552] Processing DeleteTablet for tablet b652a8a80246443491551e76e68d2a8b with delete_type TABLET_DATA_DELETED (replaced by tablet 18e119bc44874aecb4e83c8d6040c8cd) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.837168 1982 ts_tablet_manager.cc:1950] T d768c45bc02b4d108798efbf3f6aaf7d P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:20.837239 880 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a2143b5508445492b970d2588cf1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.837524 884 tablet_service.cc:1552] Processing DeleteTablet for tablet a0279992d1f2419ab1caae116a0391b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.837574 888 tablet_service.cc:1552] Processing DeleteTablet for tablet a10663bd306d40e7a6375d06d69be22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.837536 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 52daf1204bf142e7b82852bdec6582fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.838141 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet c8e33ba1149a44c294885d76f676160a
I20251214 20:52:20.838201 1982 tablet_replica.cc:333] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:20.838794 451 catalog_manager.cc:6724] ReplaceTablet: tablet c8e33ba1149a44c294885d76f676160a deleted and replaced by tablet 1eff8a23d48c43e2bdbfb34f4bab6edf
I20251214 20:52:20.839212 1982 raft_consensus.cc:2243] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.839399 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet c8e33ba1149a44c294885d76f676160a with delete_type TABLET_DATA_DELETED (replaced by tablet 1eff8a23d48c43e2bdbfb34f4bab6edf) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.840072 1982 raft_consensus.cc:2272] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.840147 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 0cafed2861e94cd9b0d10d33864ba45d
I20251214 20:52:20.841162 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cafed2861e94cd9b0d10d33864ba45d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.844141 1982 ts_tablet_manager.cc:1916] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.848037 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet d768c45bc02b4d108798efbf3f6aaf7d (table default.Table_ca875b1006554ba6845c0cf23671a8da [id=8c192689a28f4314bb5608c08ea21a2e]) successfully deleted
W20251214 20:52:20.850811 450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:50727 (request call id 182) took 893 ms (client timeout 1000 ms). Trace:
I20251214 20:52:20.852694 757 tablet_service.cc:1552] Processing DeleteTablet for tablet c8e33ba1149a44c294885d76f676160a with delete_type TABLET_DATA_DELETED (replaced by tablet 1eff8a23d48c43e2bdbfb34f4bab6edf) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:20.852825 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.853121 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff58f321124d12b5330dca185d426e with delete_type TABLET_DATA_DELETED (replaced by tablet 42bdcecba5534294aca1eb419f18e9d2) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.855787 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cafed2861e94cd9b0d10d33864ba45d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:20.856850 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 1dca369dc5f54a4ea1783e334c5670d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dca369dc5f54a4ea1783e334c5670d6 already in progress: creating tablet
I20251214 20:52:20.857088 892 tablet_service.cc:1552] Processing DeleteTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.857375 883 tablet_service.cc:1552] Processing DeleteTablet for tablet bb698ff30fcc47d09000cc68d2c5225b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.857545 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.857823 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_4d5d73b55b6245bf8594a4fc79f02adc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.858364 989 consensus_peers.cc:597] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:20.858595 1203 tablet_bootstrap.cc:492] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:20.858757 1203 ts_tablet_manager.cc:1403] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.003s
W20251214 20:52:20.858593 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d5d73b55b6245bf8594a4fc79f02adc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.859604 1203 raft_consensus.cc:359] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.859802 1203 raft_consensus.cc:385] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.859866 1203 raft_consensus.cc:740] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.862135 1856 ts_tablet_manager.cc:1916] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.863138 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 40a9a6192c744282ab5672cbe15fd299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.864694 1963 raft_consensus.cc:493] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.864867 1963 raft_consensus.cc:515] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.864984 891 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b7149a436a4cab80b39adccf8567f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.865526 1963 leader_election.cc:290] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.865501 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d8953672be144941ab3319e5477b8c57" } modify_external_catalogs: true
I20251214 20:52:20.865710 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d8953672be144941ab3319e5477b8c57" } modify_external_catalogs: true
I20251214 20:52:20.866951 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6c0f445cd7e44769d79ad5566fd8d97" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:20.867178 1053 raft_consensus.cc:2468] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:20.867626 713 leader_election.cc:304] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:20.867908 1963 raft_consensus.cc:2804] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:20.867997 1963 raft_consensus.cc:493] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:20.868100 1963 raft_consensus.cc:3060] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.868860 1206 tablet_bootstrap.cc:492] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:20.869068 1206 ts_tablet_manager.cc:1403] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.006s
I20251214 20:52:20.869882 1206 raft_consensus.cc:359] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.870106 1206 raft_consensus.cc:385] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:20.870226 1206 raft_consensus.cc:740] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:20.870693 1963 raft_consensus.cc:515] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.871101 1206 consensus_queue.cc:260] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.871634 1206 ts_tablet_manager.cc:1434] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:20.872112 1206 tablet_bootstrap.cc:492] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:20.850942 450 rpcz_store.cc:269] 1214 20:52:19.956938 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.970119 (+ 13181us) service_pool.cc:225] Handling call
1214 20:52:20.850753 (+880634us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:20.353117 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.353223 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.353230 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:20.353233 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:20.353348 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.353364 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.353369 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:20.353448 (+ 79us) tablet.cc:699] Row locks acquired
1214 20:52:20.353451 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:20.353498 (+ 47us) write_op.cc:270] Start()
1214 20:52:20.353537 (+ 39us) write_op.cc:276] Timestamp: P: 1765745540353394 usec, L: 0
1214 20:52:20.353541 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.353778 (+ 237us) log.cc:844] Serialized 356 byte log entry
1214 20:52:20.383874 (+ 30096us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.390631 (+ 6757us) write_op.cc:301] APPLY: starting
1214 20:52:20.390650 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.390692 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.390720 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.390781 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.390817 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:20.390825 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:20.390906 (+ 81us) log.cc:844] Serialized 23 byte log entry
1214 20:52:20.390975 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.390995 (+ 20us) write_op.cc:454] Released schema lock
1214 20:52:20.391002 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:20.391084 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.391088 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1214 20:52:20.682901 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.682984 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.682988 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:20.682990 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:20.683066 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.683074 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.683077 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:20.683160 (+ 83us) tablet.cc:699] Row locks acquired
1214 20:52:20.683162 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:20.683189 (+ 27us) write_op.cc:270] Start()
1214 20:52:20.683214 (+ 25us) write_op.cc:276] Timestamp: P: 1765745540683153 usec, L: 0
1214 20:52:20.683218 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.683434 (+ 216us) log.cc:844] Serialized 356 byte log entry
1214 20:52:20.696376 (+ 12942us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.696744 (+ 368us) write_op.cc:301] APPLY: starting
1214 20:52:20.696761 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.696814 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.696848 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.696917 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.696957 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:20.696964 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:20.697051 (+ 87us) log.cc:844] Serialized 23 byte log entry
1214 20:52:20.697128 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.697149 (+ 21us) write_op.cc:454] Released schema lock
1214 20:52:20.697156 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:20.697235 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.697239 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1214 20:52:20.742998 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.743088 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.743093 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:20.743095 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:20.743193 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.743202 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.743205 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:20.743249 (+ 44us) tablet.cc:699] Row locks acquired
1214 20:52:20.743252 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:20.743279 (+ 27us) write_op.cc:270] Start()
1214 20:52:20.743302 (+ 23us) write_op.cc:276] Timestamp: P: 1765745540743229 usec, L: 0
1214 20:52:20.743305 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.743456 (+ 151us) log.cc:844] Serialized 356 byte log entry
1214 20:52:20.772448 (+ 28992us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.773681 (+ 1233us) write_op.cc:301] APPLY: starting
1214 20:52:20.773690 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.773713 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.773741 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.773769 (+ 28us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.773790 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:20.773797 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:20.773836 (+ 39us) log.cc:844] Serialized 23 byte log entry
1214 20:52:20.773872 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.773881 (+ 9us) write_op.cc:454] Released schema lock
1214 20:52:20.773885 (+ 4us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:20.773947 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.773950 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
1214 20:52:20.804542 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.804675 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.804681 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:20.804683 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:20.804856 (+ 173us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.804867 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.804872 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:20.804936 (+ 64us) tablet.cc:699] Row locks acquired
1214 20:52:20.804938 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:20.804972 (+ 34us) write_op.cc:270] Start()
1214 20:52:20.805000 (+ 28us) write_op.cc:276] Timestamp: P: 1765745540804908 usec, L: 0
1214 20:52:20.805003 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.805183 (+ 180us) log.cc:844] Serialized 356 byte log entry
1214 20:52:20.814313 (+ 9130us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.816099 (+ 1786us) write_op.cc:301] APPLY: starting
1214 20:52:20.816110 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.816133 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.816157 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.816187 (+ 30us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.816209 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:20.816215 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:20.816259 (+ 44us) log.cc:844] Serialized 23 byte log entry
1214 20:52:20.816300 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.816311 (+ 11us) write_op.cc:454] Released schema lock
1214 20:52:20.816315 (+ 4us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:20.816384 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.816388 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":594,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":6695,"apply.run_cpu_time_us":447,"apply.run_wall_time_us":447,"num_ops":1,"prepare.queue_time_us":69611,"prepare.run_cpu_time_us":788,"prepare.run_wall_time_us":933,"replication_time_us":30292}],["op",{"apply.queue_time_us":310,"apply.run_cpu_time_us":486,"apply.run_wall_time_us":485,"num_ops":1,"prepare.queue_time_us":11835,"prepare.run_cpu_time_us":613,"prepare.run_wall_time_us":613,"raft.queue_time_us":103,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":13128,"spinlock_wait_cycles":768}],["op",{"apply.queue_time_us":1197,"apply.run_cpu_time_us":257,"apply.run_wall_time_us":257,"num_ops":1,"prepare.queue_time_us":31512,"prepare.run_cpu_time_us":2215,"prepare.run_wall_time_us":6956,"raft.queue_time_us":2309,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":3,"replication_time_us":29128,"thread_start_us":6374,"threads_started":1}],["op",{"apply.queue_time_us":1751,"apply.run_cpu_time_us":279,"apply.run_wall_time_us":278,"num_ops":1,"prepare.queue_time_us":24267,"prepare.run_cpu_time_us":695,"prepare.run_wall_time_us":742,"raft.queue_time_us":617,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":2,"replication_time_us":9285}]]}
I20251214 20:52:20.875717 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_cae5b2188af54d068db115a0745ee866" } new_table_name: "default.Table_9151bdce51684c39901786aef1c63ec5" modify_external_catalogs: true
I20251214 20:52:20.877547 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_55fe32e16f364e58afdd57c80a8966ae" } new_table_name: "default.Table_76f578444d71410a93c3f78ff50b7951" modify_external_catalogs: true
I20251214 20:52:20.877934 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7c6fe3b24daca70711a1990af556 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.878258 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.878489 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.876380 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c488616dc265498088254d401a3cc7db" } modify_external_catalogs: true
I20251214 20:52:20.878958 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c488616dc265498088254d401a3cc7db" } modify_external_catalogs: true
I20251214 20:52:20.879686 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff with delete_type TABLET_DATA_DELETED (replaced by tablet 6a000951f61948ac85fdd431f3d162c2) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.880040 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af99e0bd8774a678575847a9ec2e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.880117 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 52daf1204bf142e7b82852bdec6582fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.880044 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.880390 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a7fc15882e2474bb6885070be13b1fa with delete_type TABLET_DATA_DELETED (replaced by tablet 65e79db74fa046698507c83416ee1669) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.880582 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_53273de8ee634ce9994c46fe98bba85e" } new_table_name: "default.Table_246962600e464910aa3a756037dd08e9" modify_external_catalogs: true
I20251214 20:52:20.880712 1963 leader_election.cc:290] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:20.881291 1206 tablet_bootstrap.cc:654] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:20.881458 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 9523d72fd91445539d67e1eac2a80afd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.881860 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.882725 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet f0d661a1bb034053a4634ff537eafc7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.883819 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet bb698ff30fcc47d09000cc68d2c5225b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.884073 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a2143b5508445492b970d2588cf1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.884979 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 63497e77baeb4d328be2421b5732a28a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.885864 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 2da66ec83e9140f7a605f0c86abff468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.886569 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.887307 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.888310 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet b652a8a80246443491551e76e68d2a8b with delete_type TABLET_DATA_DELETED (replaced by tablet 18e119bc44874aecb4e83c8d6040c8cd) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.890307 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6c0f445cd7e44769d79ad5566fd8d97" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:20.890411 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet c25832da5c1644329e18653000f3330e with delete_type TABLET_DATA_DELETED (replaced by tablet 0c26d1f969ed4071a2f2cf1615a17cac) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.890578 1053 raft_consensus.cc:3060] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:20.891518 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b4f582d8165439d86d9228c146dc41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.891798 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:20.892915 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d9749ec3d4b8476a9e3d18ff2d300544"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:20.893484 1053 raft_consensus.cc:2468] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
W20251214 20:52:20.894326 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9749ec3d4b8476a9e3d18ff2d300544 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.896786 713 leader_election.cc:304] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:20.897126 1963 raft_consensus.cc:2804] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:20.897747 1963 raft_consensus.cc:697] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:20.898026 1963 consensus_queue.cc:237] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.898270 895 tablet_service.cc:1552] Processing DeleteTablet for tablet c16895ca08d94b56b5c4bc9352e330ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:20.898809 712 leader_election.cc:343] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: d6c0f445cd7e44769d79ad5566fd8d97
I20251214 20:52:20.899868 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd6856b800247de896f16eee66f2827 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c3962c33a294fa98f8bd485268aee71) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:20.900556 712 leader_election.cc:343] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: d6c0f445cd7e44769d79ad5566fd8d97
I20251214 20:52:20.901345 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 324379e3217442c3addcfe58060b9ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.901608 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 2da66ec83e9140f7a605f0c86abff468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.901744 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cafed2861e94cd9b0d10d33864ba45d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.901849 895 tablet_service.cc:1552] Processing DeleteTablet for tablet c8e33ba1149a44c294885d76f676160a with delete_type TABLET_DATA_DELETED (replaced by tablet 1eff8a23d48c43e2bdbfb34f4bab6edf) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.902137 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3962c33a294fa98f8bd485268aee71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.902503 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.902810 895 tablet_service.cc:1552] Processing DeleteTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:20.903766 1982 ts_tablet_manager.cc:1929] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.903926 1982 log.cc:1199] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/f3a2143b5508445492b970d2588cf1cf
I20251214 20:52:20.904269 1982 ts_tablet_manager.cc:1950] T f3a2143b5508445492b970d2588cf1cf P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:20.905166 1982 tablet_replica.cc:333] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:20.905349 1982 raft_consensus.cc:2243] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.905680 1982 raft_consensus.cc:2272] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.906220 1982 ts_tablet_manager.cc:1916] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.907059 1856 ts_tablet_manager.cc:1929] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.907203 1856 log.cc:1199] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/84a0237e4ee34f94821d8647a400575b
I20251214 20:52:20.907581 1856 ts_tablet_manager.cc:1950] T 84a0237e4ee34f94821d8647a400575b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.912379 2031 tablet.cc:1722] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:20.912928 823 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:20.913338 2037 tablet.cc:1722] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:20.913956 2032 tablet.cc:1722] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:20.916164 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 64) took 1920 ms. Trace:
I20251214 20:52:20.916347 1856 rpcz_store.cc:276] 1214 20:52:18.994782 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.997147 (+ 2365us) service_pool.cc:225] Handling call
1214 20:52:20.801855 (+1804708us) ts_tablet_manager.cc:1151] Deleting tablet 84a0237e4ee34f94821d8647a400575b
1214 20:52:20.801866 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.886808 (+ 84942us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.906929 (+ 20121us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.915531 (+ 8602us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1774223}
I20251214 20:52:20.917819 2034 tablet.cc:1722] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:20.920176 1977 tablet_replica.cc:333] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:20.920208 961 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:20.922717 1977 raft_consensus.cc:2243] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.922978 1977 raft_consensus.cc:2272] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.923743 1977 ts_tablet_manager.cc:1916] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.916899 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 77) took 1920 ms. Trace:
I20251214 20:52:20.927472 1856 rpcz_store.cc:276] 1214 20:52:18.996536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.999406 (+ 2870us) service_pool.cc:225] Handling call
1214 20:52:20.916686 (+1917280us) ts_tablet_manager.cc:1151] Deleting tablet f84e9dfc23d34eb0a379e8ac5a442e59
1214 20:52:20.916698 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.916859 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1888424}
I20251214 20:52:20.928550 1856 tablet_replica.cc:333] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:20.929316 1856 raft_consensus.cc:2243] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:20.930225 1856 raft_consensus.cc:2272] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.935865 1856 ts_tablet_manager.cc:1916] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.939067 2035 tablet.cc:1722] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:20.945519 1982 ts_tablet_manager.cc:1929] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.945705 1982 log.cc:1199] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/f0d661a1bb034053a4634ff537eafc7d
I20251214 20:52:20.946151 1982 ts_tablet_manager.cc:1950] T f0d661a1bb034053a4634ff537eafc7d P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:20.946445 1202 tablet_bootstrap.cc:492] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:20.948274 2042 tablet.cc:1722] T 3fa0317e999b483788e5c5652e6e83de P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:20.949476 2043 tablet.cc:1722] T 3fa0317e999b483788e5c5652e6e83de P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:20.950310 1203 consensus_queue.cc:260] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.951382 1203 ts_tablet_manager.cc:1434] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.093s user 0.003s sys 0.000s
I20251214 20:52:20.952752 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet f3a2143b5508445492b970d2588cf1cf (table default.Table_d653f9c533784976ac955f90dc39eb56 [id=295a34df92b44b738e3aa3d1d8195673]) successfully deleted
I20251214 20:52:20.953228 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet f0d661a1bb034053a4634ff537eafc7d (table default.Table_4e95a2afe6004b6eb41370206c6faa1f [id=4b6133c53ae64267993fbf56afa38f90]) successfully deleted
I20251214 20:52:20.958369 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 84a0237e4ee34f94821d8647a400575b (table default.Table_2162a5b765254842bb0b7e4ba7f66136 [id=fa79ab98ae4f405d830ed263d3c0b3c1]) successfully deleted
W20251214 20:52:20.959061 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f84e9dfc23d34eb0a379e8ac5a442e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f84e9dfc23d34eb0a379e8ac5a442e59 already in progress: creating tablet
I20251214 20:52:20.960538 1982 tablet_replica.cc:333] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
W20251214 20:52:20.963225 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet bb698ff30fcc47d09000cc68d2c5225b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb698ff30fcc47d09000cc68d2c5225b already in progress: creating tablet
I20251214 20:52:20.963568 1982 raft_consensus.cc:2243] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:20.963822 1982 raft_consensus.cc:2272] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:20.967031 2034 tablet.cc:1722] T 3fa0317e999b483788e5c5652e6e83de P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:20.977409 912 raft_consensus.cc:1275] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Refusing update from remote peer 63848a9fb78a4013a6f1657621272d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251214 20:52:20.978220 2037 tablet.cc:1722] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:20.979516 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_9a53d238203b418eabb29093faef194c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.980406 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a53d238203b418eabb29093faef194c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.980522 1882 consensus_queue.cc:1048] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:20.983740 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d20c0b0d957a47c3985941ceda684666" } new_table_name: "default.Table_99e702b2f3fc436db82c793284a1670c" modify_external_catalogs: true
W20251214 20:52:20.985752 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e31615d1277412eb8bd80451f2c4366 consensus state has no leader
I20251214 20:52:20.986742 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_de80fc127aa549ca931866a310858ddb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:20.989265 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de80fc127aa549ca931866a310858ddb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:20.989624 1856 ts_tablet_manager.cc:1929] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:20.989818 1856 log.cc:1199] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/cd88d1e95ee04ef596863e1ae3687de0
I20251214 20:52:20.990275 1856 ts_tablet_manager.cc:1950] T cd88d1e95ee04ef596863e1ae3687de0 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:20.994956 2048 raft_consensus.cc:493] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:20.996779 2048 raft_consensus.cc:515] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:20.997360 1982 ts_tablet_manager.cc:1916] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:20.997380 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 324379e3217442c3addcfe58060b9ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.000515 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e16fe751f10f4616a00d0af811998729" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:21.002365 851 leader_election.cc:343] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.003302 2048 leader_election.cc:290] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:21.007087 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e16fe751f10f4616a00d0af811998729" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:21.007938 848 leader_election.cc:343] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.008184 848 leader_election.cc:304] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:21.008922 2048 raft_consensus.cc:2749] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:21.012609 2035 tablet.cc:1722] T 07a8662a634a4befbd2d6251000d2059 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:21.017772 2031 tablet.cc:1722] T 07a8662a634a4befbd2d6251000d2059 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:21.018106 2032 tablet.cc:1722] T afd51f330eb649b998a22a343d993b30 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:21.024196 1963 raft_consensus.cc:493] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.024453 1963 raft_consensus.cc:515] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.025472 1963 leader_election.cc:290] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:21.025856 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f9def8e4fd449599820a63845397750 has no consensus state
I20251214 20:52:21.026391 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_5577714724264f51932f490a337aa8d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:21.028683 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c0f49e5d78240a296e47eccb3218e4f" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:21.029006 904 raft_consensus.cc:2468] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:21.029868 712 leader_election.cc:304] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:21.030550 1964 raft_consensus.cc:2804] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:21.030616 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c2048db412744e2da280a49ec3d679b8" } modify_external_catalogs: true
I20251214 20:52:21.030989 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c2048db412744e2da280a49ec3d679b8" } modify_external_catalogs: true
I20251214 20:52:21.031222 1964 raft_consensus.cc:493] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:21.031340 1964 raft_consensus.cc:3060] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.033901 1964 raft_consensus.cc:515] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.034544 1964 leader_election.cc:290] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.037956 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd6856b800247de896f16eee66f2827 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c3962c33a294fa98f8bd485268aee71) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.038980 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.039983 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d74d3abc45a41969289c0a0892bfa04 with delete_type TABLET_DATA_DELETED (replaced by tablet 727e7b61ae3942098712720d344c5310) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.041222 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet ca263a80702f4f64aa7c872c2be08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.043040 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9c0f49e5d78240a296e47eccb3218e4f" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:21.043287 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 65) took 1996 ms. Trace:
I20251214 20:52:21.043408 1856 rpcz_store.cc:276] 1214 20:52:18.994859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.997183 (+ 2324us) service_pool.cc:225] Handling call
1214 20:52:20.928461 (+1931278us) ts_tablet_manager.cc:1151] Deleting tablet cd88d1e95ee04ef596863e1ae3687de0
1214 20:52:20.928473 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:20.949061 (+ 20588us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.989451 (+ 40390us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:20.991118 (+ 1667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1898734}
I20251214 20:52:21.043715 912 raft_consensus.cc:3060] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.044004 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 67) took 2048 ms. Trace:
I20251214 20:52:21.044108 1856 rpcz_store.cc:276] 1214 20:52:18.995424 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.995490 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:21.043801 (+2048311us) ts_tablet_manager.cc:1151] Deleting tablet 47e813373fbd482298312ead4f037c01
1214 20:52:21.043814 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.043938 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2013292}
I20251214 20:52:21.044421 1856 tablet_replica.cc:333] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.044660 1856 raft_consensus.cc:2243] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.044894 1856 raft_consensus.cc:2272] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.045599 1856 ts_tablet_manager.cc:1916] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.051159 2031 tablet.cc:1722] T afd51f330eb649b998a22a343d993b30 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251214 20:52:21.056313 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5577714724264f51932f490a337aa8d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:21.057412 912 raft_consensus.cc:2468] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:21.058106 712 leader_election.cc:304] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:21.058486 1964 raft_consensus.cc:2804] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:21.059181 1964 raft_consensus.cc:697] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:21.059566 1964 consensus_queue.cc:237] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.075781 1856 ts_tablet_manager.cc:1929] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.075976 1856 log.cc:1199] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/8faa7fd7315f485c82aee38bcf4cd8fc
I20251214 20:52:21.076418 1856 ts_tablet_manager.cc:1950] T 8faa7fd7315f485c82aee38bcf4cd8fc P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.077334 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 61) took 2082 ms. Trace:
I20251214 20:52:21.077471 1856 rpcz_store.cc:276] 1214 20:52:18.994475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.997124 (+ 2649us) service_pool.cc:225] Handling call
1214 20:52:21.044325 (+2047201us) ts_tablet_manager.cc:1151] Deleting tablet 8faa7fd7315f485c82aee38bcf4cd8fc
1214 20:52:21.044334 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.072510 (+ 28176us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.075630 (+ 3120us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.077292 (+ 1662us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2013697}
I20251214 20:52:21.079797 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 68) took 2084 ms. Trace:
I20251214 20:52:21.079931 1856 rpcz_store.cc:276] 1214 20:52:18.995537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.995577 (+ 40us) service_pool.cc:225] Handling call
1214 20:52:21.079582 (+2084005us) ts_tablet_manager.cc:1151] Deleting tablet 5da9deeedce0421c9aeaebd71ba11b1b
1214 20:52:21.079595 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.079759 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2045550}
I20251214 20:52:21.080334 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 69) took 2084 ms. Trace:
I20251214 20:52:21.080451 1856 rpcz_store.cc:276] 1214 20:52:18.995642 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.995692 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:21.080190 (+2084498us) ts_tablet_manager.cc:1151] Deleting tablet ec85857f047e44d78835305ddcb70768
1214 20:52:21.080198 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.080297 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2047364}
I20251214 20:52:21.080724 1856 tablet_replica.cc:333] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.080965 1856 raft_consensus.cc:2243] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.081149 1856 raft_consensus.cc:2272] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.081807 1856 ts_tablet_manager.cc:1916] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.083909 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b22043d81b493fb12116422b1d3fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet ba7952290e2c4746802cffeb14971da0) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.088218 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet cd88d1e95ee04ef596863e1ae3687de0 (table default.Table_bda8a86e13ba427e94e8d85a318281ad [id=2018af6df0b448bd8996691cb5a1f62c]) successfully deleted
I20251214 20:52:21.088945 1856 ts_tablet_manager.cc:1929] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:21.088963 1982 ts_tablet_manager.cc:1929] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:21.089115 1856 log.cc:1199] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/569a02142c71427d91868b02695ae12b
I20251214 20:52:21.089147 1982 log.cc:1199] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/324379e3217442c3addcfe58060b9ad7
I20251214 20:52:21.089488 1856 ts_tablet_manager.cc:1950] T 569a02142c71427d91868b02695ae12b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.089838 1202 tablet_bootstrap.cc:654] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.090824 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 60) took 2096 ms. Trace:
I20251214 20:52:21.090945 1856 rpcz_store.cc:276] 1214 20:52:18.994379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.994443 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:21.080655 (+2086212us) ts_tablet_manager.cc:1151] Deleting tablet 569a02142c71427d91868b02695ae12b
1214 20:52:21.080662 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.085327 (+ 4665us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.088799 (+ 3472us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.090764 (+ 1965us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2047739}
I20251214 20:52:21.091328 1856 tablet_replica.cc:333] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.091580 1203 tablet_bootstrap.cc:492] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:21.091555 1856 raft_consensus.cc:2243] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.092128 1856 raft_consensus.cc:2272] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.092794 1856 ts_tablet_manager.cc:1916] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.093876 1203 tablet_bootstrap.cc:654] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:21.095326 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 47e813373fbd482298312ead4f037c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e813373fbd482298312ead4f037c01 already in progress: creating tablet
I20251214 20:52:21.096211 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 8faa7fd7315f485c82aee38bcf4cd8fc (table default.Table_1d198a3975c24b84aaeb82c266c746b8 [id=bc62e2586be140efbc2bfb5a3b5baaa2]) successfully deleted
W20251214 20:52:21.098989 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ec85857f047e44d78835305ddcb70768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec85857f047e44d78835305ddcb70768 already in progress: creating tablet
I20251214 20:52:21.099591 1856 ts_tablet_manager.cc:1929] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.099746 1856 log.cc:1199] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:21.100512 1856 ts_tablet_manager.cc:1950] T 649f5bd67cbb4f61b7066269f0bf9266 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:21.100750 991 consensus_peers.cc:597] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a34eb8fa3e34427495fa5985b7ba81bf. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:21.101420 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 80) took 2104 ms. Trace:
I20251214 20:52:21.101554 1856 rpcz_store.cc:276] 1214 20:52:18.996759 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.024088 (+ 27329us) service_pool.cc:225] Handling call
1214 20:52:21.091244 (+2067156us) ts_tablet_manager.cc:1151] Deleting tablet 649f5bd67cbb4f61b7066269f0bf9266
1214 20:52:21.091253 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.096091 (+ 4838us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.099440 (+ 3349us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.101372 (+ 1932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2058235}
I20251214 20:52:21.102020 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 59) took 2107 ms. Trace:
I20251214 20:52:21.102151 1856 rpcz_store.cc:276] 1214 20:52:18.994231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.997078 (+ 2847us) service_pool.cc:225] Handling call
1214 20:52:21.101862 (+2104784us) ts_tablet_manager.cc:1151] Deleting tablet b315744c8cd043c5a8bdceec73a7e28b
1214 20:52:21.101871 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.101984 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2068228}
I20251214 20:52:21.102588 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 76) took 2106 ms. Trace:
I20251214 20:52:21.103673 1856 rpcz_store.cc:276] 1214 20:52:18.996418 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.996469 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:21.102356 (+2105887us) ts_tablet_manager.cc:1151] Deleting tablet ee7d709316b24b8eb223a838710c2979
1214 20:52:21.102365 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.102534 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2068396}
I20251214 20:52:21.103919 1964 raft_consensus.cc:493] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.104157 1964 raft_consensus.cc:515] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.104395 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 74) took 2108 ms. Trace:
I20251214 20:52:21.104512 1856 rpcz_store.cc:276] 1214 20:52:18.996222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.996267 (+ 45us) service_pool.cc:225] Handling call
1214 20:52:21.104218 (+2107951us) ts_tablet_manager.cc:1151] Deleting tablet 8d9f74fb2c6244a6887e3c922262bbab
1214 20:52:21.104228 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.104365 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2070147}
I20251214 20:52:21.105125 1856 tablet_replica.cc:333] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.105505 1856 raft_consensus.cc:2243] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.105881 1856 raft_consensus.cc:2272] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.106564 1856 ts_tablet_manager.cc:1916] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.110132 1964 leader_election.cc:290] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.114351 1856 ts_tablet_manager.cc:1929] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251214 20:52:21.114591 712 leader_election.cc:343] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: a34eb8fa3e34427495fa5985b7ba81bf
I20251214 20:52:21.121680 1982 ts_tablet_manager.cc:1950] T 324379e3217442c3addcfe58060b9ad7 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.122896 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 76) took 1089 ms. Trace:
I20251214 20:52:21.123283 1982 rpcz_store.cc:276] 1214 20:52:20.033148 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.033877 (+ 729us) service_pool.cc:225] Handling call
1214 20:52:20.960443 (+926566us) ts_tablet_manager.cc:1151] Deleting tablet 324379e3217442c3addcfe58060b9ad7
1214 20:52:20.960451 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.049503 (+ 89052us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.088817 (+ 39314us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.122807 (+ 33990us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":926377}
I20251214 20:52:21.124704 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 324379e3217442c3addcfe58060b9ad7 (table default.Table_ff05c589ffd8444daa99c956e92c37e6 [id=16a8869e1ba641438d0836f6f204414f]) successfully deleted
I20251214 20:52:21.125923 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6f66a268b27e4f9b8d09597224149553" } modify_external_catalogs: true
I20251214 20:52:21.125211 1982 tablet_replica.cc:333] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.132207 1982 raft_consensus.cc:2243] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.134857 1982 raft_consensus.cc:2272] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.135643 1982 ts_tablet_manager.cc:1916] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.135911 1856 log.cc:1199] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/60ae3db968f6412aa0c3952fafe65f88
I20251214 20:52:21.136528 1856 ts_tablet_manager.cc:1950] T 60ae3db968f6412aa0c3952fafe65f88 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.135712 2046 tablet.cc:1722] T afd51f330eb649b998a22a343d993b30 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:21.139344 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 70) took 2143 ms. Trace:
I20251214 20:52:21.139496 1856 rpcz_store.cc:276] 1214 20:52:18.995735 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.997230 (+ 1495us) service_pool.cc:225] Handling call
1214 20:52:21.105065 (+2107835us) ts_tablet_manager.cc:1151] Deleting tablet 60ae3db968f6412aa0c3952fafe65f88
1214 20:52:21.105073 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.109933 (+ 4860us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.114215 (+ 4282us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.139271 (+ 25056us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2070584}
I20251214 20:52:21.140010 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 75) took 2143 ms. Trace:
I20251214 20:52:21.140105 1856 rpcz_store.cc:276] 1214 20:52:18.996333 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.997302 (+ 969us) service_pool.cc:225] Handling call
1214 20:52:21.139797 (+2142495us) ts_tablet_manager.cc:1151] Deleting tablet b2e83bca8ed642b6a1b705ef70fde27f
1214 20:52:21.139808 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.139947 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2104573}
I20251214 20:52:21.140398 1856 tablet_replica.cc:333] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.140620 1856 raft_consensus.cc:2243] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.141022 1856 raft_consensus.cc:2272] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.141695 1856 ts_tablet_manager.cc:1916] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.157171 1964 raft_consensus.cc:493] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.157397 1964 raft_consensus.cc:515] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.158089 1964 leader_election.cc:290] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.159001 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a10663bd306d40e7a6375d06d69be22d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:21.169153 1982 ts_tablet_manager.cc:1929] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.172686 1977 ts_tablet_manager.cc:1929] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.173032 1977 log.cc:1199] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/472e4a5990e64d02b1d52dc6e86d7190
I20251214 20:52:21.174139 1977 ts_tablet_manager.cc:1950] T 472e4a5990e64d02b1d52dc6e86d7190 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:21.175024 1856 ts_tablet_manager.cc:1929] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.175314 1982 log.cc:1199] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/6bd6856b800247de896f16eee66f2827
I20251214 20:52:21.175879 1982 ts_tablet_manager.cc:1950] T 6bd6856b800247de896f16eee66f2827 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.176290 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 77) took 1141 ms. Trace:
I20251214 20:52:21.176474 1982 rpcz_store.cc:276] 1214 20:52:20.034334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.041567 (+ 7233us) service_pool.cc:225] Handling call
1214 20:52:21.124847 (+1083280us) ts_tablet_manager.cc:1151] Deleting tablet 6bd6856b800247de896f16eee66f2827
1214 20:52:21.124860 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.159533 (+ 34673us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.168974 (+ 9441us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.176245 (+ 7271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1083007}
I20251214 20:52:21.177085 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 6bd6856b800247de896f16eee66f2827 (table default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]) successfully deleted
I20251214 20:52:21.177163 1982 tablet_replica.cc:333] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.177377 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 569a02142c71427d91868b02695ae12b (table default.Table_7b427a00182f4e059423bf0e18859b90 [id=17c89c03be10405f834504b803e594c7]) successfully deleted
I20251214 20:52:21.177580 1982 raft_consensus.cc:2243] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.177851 1982 raft_consensus.cc:2272] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.178561 1982 ts_tablet_manager.cc:1916] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.178908 1206 tablet_bootstrap.cc:492] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.180447 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 649f5bd67cbb4f61b7066269f0bf9266 (table default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]) successfully deleted
W20251214 20:52:21.179860 713 leader_election.cc:343] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 9c0f49e5d78240a296e47eccb3218e4f
W20251214 20:52:21.180940 713 leader_election.cc:343] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 9c0f49e5d78240a296e47eccb3218e4f
W20251214 20:52:21.181272 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b315744c8cd043c5a8bdceec73a7e28b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b315744c8cd043c5a8bdceec73a7e28b already in progress: creating tablet
I20251214 20:52:21.181655 1206 ts_tablet_manager.cc:1403] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.310s user 0.018s sys 0.036s
I20251214 20:52:21.185403 1206 raft_consensus.cc:359] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.187870 1206 raft_consensus.cc:385] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.188251 1206 raft_consensus.cc:740] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.188767 1206 consensus_queue.cc:260] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.191017 1206 ts_tablet_manager.cc:1434] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:21.195039 1206 tablet_bootstrap.cc:492] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.195915 1202 tablet_bootstrap.cc:492] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:21.196349 1202 ts_tablet_manager.cc:1403] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.332s user 0.005s sys 0.000s
I20251214 20:52:21.197196 1206 tablet_bootstrap.cc:654] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.198369 1202 raft_consensus.cc:359] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.200449 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d80c747f5a704a608b5e28b5e5d247b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:21.201352 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d80c747f5a704a608b5e28b5e5d247b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:21.203509 439 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2f9def8e4fd449599820a63845397750 (table default.Table_99e702b2f3fc436db82c793284a1670c [id=a9ff75b0f47f4601b2d489c3e3d6fce0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2f9def8e4fd449599820a63845397750 consensus state has no leader
I20251214 20:52:21.202404 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6f66a268b27e4f9b8d09597224149553" } modify_external_catalogs: true
I20251214 20:52:21.205013 1856 log.cc:1199] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/40a9a6192c744282ab5672cbe15fd299
I20251214 20:52:21.208742 1856 ts_tablet_manager.cc:1950] T 40a9a6192c744282ab5672cbe15fd299 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:21.208825 446 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 2accf8a160584c82bb695a52e3f2b05c (table default.Table_b4334a1d99a54c2eabe2fb909563313c [id=d31b208e2e284575ad429878129a28f5], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 2accf8a160584c82bb695a52e3f2b05c consensus state has no leader
I20251214 20:52:21.209862 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 73) took 2213 ms. Trace:
I20251214 20:52:21.210008 1856 rpcz_store.cc:276] 1214 20:52:18.996108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.997279 (+ 1171us) service_pool.cc:225] Handling call
1214 20:52:21.140302 (+2143023us) ts_tablet_manager.cc:1151] Deleting tablet 40a9a6192c744282ab5672cbe15fd299
1214 20:52:21.140311 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.170082 (+ 29771us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.174877 (+ 4795us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.209810 (+ 34933us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2104562}
I20251214 20:52:21.210398 1856 tablet_replica.cc:333] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.210649 1856 raft_consensus.cc:2243] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.211061 1856 raft_consensus.cc:2272] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.216346 757 tablet_service.cc:1505] Processing CreateTablet for tablet 76472ce9005b4bd2bbc51278816e8ae0 (DEFAULT_TABLE table=default.Table_e74a0c96be654f4c808071da1ae734a0 [id=4b3f6c8d8cce4f7a96b2c677df6469ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.216907 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76472ce9005b4bd2bbc51278816e8ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.182588 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet a063083f54ad49d2a842fa080d44ed85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.201221 1102 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:21.179288 1203 tablet_bootstrap.cc:492] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:21.224045 1203 ts_tablet_manager.cc:1403] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.133s user 0.004s sys 0.000s
I20251214 20:52:21.225044 1206 tablet_bootstrap.cc:492] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.225030 1203 raft_consensus.cc:359] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.225221 1203 raft_consensus.cc:385] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.225320 1203 raft_consensus.cc:740] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.225503 1206 ts_tablet_manager.cc:1403] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.002s
I20251214 20:52:21.225543 1203 consensus_queue.cc:260] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.226200 1203 ts_tablet_manager.cc:1434] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:21.227145 1206 raft_consensus.cc:359] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.227380 1206 raft_consensus.cc:385] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.227487 1206 raft_consensus.cc:740] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.227756 1206 consensus_queue.cc:260] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.228442 1206 ts_tablet_manager.cc:1434] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251214 20:52:21.234230 743 tablet_service.cc:1505] Processing CreateTablet for tablet 4501fd5566a34e2a82009cc9ce6bdd37 (DEFAULT_TABLE table=default.Table_a941092a7b2944bfa37073939660b587 [id=14f12b293a274ebeaf5bfefdff5b78cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.236721 1856 ts_tablet_manager.cc:1916] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.236934 1206 tablet_bootstrap.cc:492] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.234493 895 tablet_service.cc:1505] Processing CreateTablet for tablet 76472ce9005b4bd2bbc51278816e8ae0 (DEFAULT_TABLE table=default.Table_e74a0c96be654f4c808071da1ae734a0 [id=4b3f6c8d8cce4f7a96b2c677df6469ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.240061 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76472ce9005b4bd2bbc51278816e8ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.238145 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4501fd5566a34e2a82009cc9ce6bdd37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.242530 1203 tablet_bootstrap.cc:492] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:21.243223 1206 tablet_bootstrap.cc:654] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.238885 891 tablet_service.cc:1505] Processing CreateTablet for tablet 4501fd5566a34e2a82009cc9ce6bdd37 (DEFAULT_TABLE table=default.Table_a941092a7b2944bfa37073939660b587 [id=14f12b293a274ebeaf5bfefdff5b78cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.244108 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4501fd5566a34e2a82009cc9ce6bdd37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.245595 1203 tablet_bootstrap.cc:654] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.199260 1202 raft_consensus.cc:385] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.255877 1202 raft_consensus.cc:740] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.256292 1202 consensus_queue.cc:260] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.257000 1202 ts_tablet_manager.cc:1434] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.060s user 0.001s sys 0.001s
I20251214 20:52:21.257021 1856 ts_tablet_manager.cc:1929] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.257181 1856 log.cc:1199] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/4b86f4adaf4540f48a4f42bc45e259e5
I20251214 20:52:21.260075 1982 ts_tablet_manager.cc:1929] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:21.270370 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 472e4a5990e64d02b1d52dc6e86d7190 (table default.Table_2acb51a56a60469e941100c99494a99b [id=b16533ff3115482d9357bd88bc1406cd]) successfully deleted
I20251214 20:52:21.261075 1206 tablet_bootstrap.cc:492] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.270825 1982 log.cc:1199] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/bd8226b431c6456885e928880938969d
I20251214 20:52:21.270937 1206 ts_tablet_manager.cc:1403] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.008s
I20251214 20:52:21.271420 1982 ts_tablet_manager.cc:1950] T bd8226b431c6456885e928880938969d P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
W20251214 20:52:21.271453 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet d6c0f445cd7e44769d79ad5566fd8d97 because the tablet was not found. No further retry: Not found: Tablet not found: d6c0f445cd7e44769d79ad5566fd8d97
W20251214 20:52:21.271677 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 604b7c6fe3b24daca70711a1990af556 because the tablet was not found. No further retry: Not found: Tablet not found: 604b7c6fe3b24daca70711a1990af556
I20251214 20:52:21.271863 1206 raft_consensus.cc:359] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.272068 1206 raft_consensus.cc:385] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.272138 1206 raft_consensus.cc:740] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.272403 1206 consensus_queue.cc:260] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:21.272500 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2b48bb260e854beb9ec61ce9c79b46c2 because the tablet was not found. No further retry: Not found: Tablet not found: 2b48bb260e854beb9ec61ce9c79b46c2
I20251214 20:52:21.272444 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 78) took 1230 ms. Trace:
W20251214 20:52:21.272701 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 16816fa6b8c34207bb929147910b93cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16816fa6b8c34207bb929147910b93cb already in progress: creating tablet
I20251214 20:52:21.272594 1982 rpcz_store.cc:276] 1214 20:52:20.042098 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.043638 (+ 1540us) service_pool.cc:225] Handling call
1214 20:52:21.177081 (+1133443us) ts_tablet_manager.cc:1151] Deleting tablet bd8226b431c6456885e928880938969d
1214 20:52:21.177092 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.181842 (+ 4750us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.259562 (+ 77720us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.272356 (+ 12794us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133237}
I20251214 20:52:21.272981 1206 ts_tablet_manager.cc:1434] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:21.273011 1982 tablet_replica.cc:333] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.273250 1982 raft_consensus.cc:2243] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.273319 1206 tablet_bootstrap.cc:492] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.273717 1982 raft_consensus.cc:2272] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:21.273967 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 9523d72fd91445539d67e1eac2a80afd because the tablet was not found. No further retry: Not found: Tablet not found: 9523d72fd91445539d67e1eac2a80afd
W20251214 20:52:21.274160 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3bbb28259c1442ae94f4eff76c25c539 because the tablet was not found. No further retry: Not found: Tablet not found: 3bbb28259c1442ae94f4eff76c25c539
W20251214 20:52:21.274350 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a34eb8fa3e34427495fa5985b7ba81bf because the tablet was not found. No further retry: Not found: Tablet not found: a34eb8fa3e34427495fa5985b7ba81bf
I20251214 20:52:21.274664 1982 ts_tablet_manager.cc:1916] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:21.274731 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet b652a8a80246443491551e76e68d2a8b because the tablet was not found. No further retry: Not found: Tablet not found: b652a8a80246443491551e76e68d2a8b
W20251214 20:52:21.275862 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet f3a2143b5508445492b970d2588cf1cf because the tablet was not found. No further retry: Not found: Tablet not found: f3a2143b5508445492b970d2588cf1cf
W20251214 20:52:21.276160 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a0279992d1f2419ab1caae116a0391b2 because the tablet was not found. No further retry: Not found: Tablet not found: a0279992d1f2419ab1caae116a0391b2
I20251214 20:52:21.276330 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet bd8226b431c6456885e928880938969d (table default.Table_1917701c01bf48c7b2bfc335d05f6db0 [id=6fba2831971a4bffac786611b54a6468]) successfully deleted
I20251214 20:52:21.276760 1206 tablet_bootstrap.cc:654] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.277076 1856 ts_tablet_manager.cc:1950] T 4b86f4adaf4540f48a4f42bc45e259e5 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.280988 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 84) took 2248 ms. Trace:
I20251214 20:52:21.286952 743 tablet_service.cc:1505] Processing CreateTablet for tablet d2688515bd1441fbad7516f8343ca775 (DEFAULT_TABLE table=default.Table_d2beb8dc1b054cada5e4c2ec9fabc8d4 [id=03997e2eab154ef18b2e3774b5984764]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.283375 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 84a0237e4ee34f94821d8647a400575b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.283831 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.284155 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.284646 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 69f6108792fa4c2a9744e3248c76251f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.284963 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet ec85857f047e44d78835305ddcb70768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.287498 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2688515bd1441fbad7516f8343ca775. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.285111 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.285301 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.285343 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet ba50e32f7e724868bb891045a939e362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.287963 1856 rpcz_store.cc:276] 1214 20:52:19.032612 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.032784 (+ 172us) service_pool.cc:225] Handling call
1214 20:52:21.210313 (+2177529us) ts_tablet_manager.cc:1151] Deleting tablet 4b86f4adaf4540f48a4f42bc45e259e5
1214 20:52:21.210324 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.253724 (+ 43400us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.256847 (+ 3123us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.280913 (+ 24066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2174137}
I20251214 20:52:21.288523 757 tablet_service.cc:1505] Processing CreateTablet for tablet f7f06eded6c340c58101fcc644ebcf50 (DEFAULT_TABLE table=default.Table_8163ed145f64447c8eb6fc0f8d0a1d4d [id=c4c61d0a0bdf4db8bd6011c151d6b87c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.289175 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f06eded6c340c58101fcc644ebcf50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.296398 1202 tablet_bootstrap.cc:492] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:21.297624 1206 tablet_bootstrap.cc:492] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.297842 1206 ts_tablet_manager.cc:1403] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.012s
I20251214 20:52:21.298629 1202 tablet_bootstrap.cc:654] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.298746 1206 raft_consensus.cc:359] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.298954 1206 raft_consensus.cc:385] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.299059 1206 raft_consensus.cc:740] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.299335 1206 consensus_queue.cc:260] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.297631 895 tablet_service.cc:1505] Processing CreateTablet for tablet d2688515bd1441fbad7516f8343ca775 (DEFAULT_TABLE table=default.Table_d2beb8dc1b054cada5e4c2ec9fabc8d4 [id=03997e2eab154ef18b2e3774b5984764]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:21.299968 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:46855 (request call id 161) took 2135 ms (client timeout 1000 ms). Trace:
I20251214 20:52:21.275635 1977 tablet_replica.cc:333] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
W20251214 20:52:21.300087 446 rpcz_store.cc:269] 1214 20:52:19.164203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.225367 (+ 61164us) service_pool.cc:225] Handling call
1214 20:52:20.034885 (+809518us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.034891 (+ 6us) write_op.cc:454] Released schema lock
1214 20:52:21.299891 (+1265000us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:20.023642 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.023760 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.023765 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:20.023767 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:20.023916 (+ 149us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.023929 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.023933 (+ 4us) tablet.cc:683] Acquiring locks for 4 operations
1214 20:52:20.024027 (+ 94us) tablet.cc:699] Row locks acquired
1214 20:52:20.024029 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:20.024064 (+ 35us) write_op.cc:270] Start()
1214 20:52:20.024091 (+ 27us) write_op.cc:276] Timestamp: P: 1765745540023953 usec, L: 0
1214 20:52:20.024094 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.024288 (+ 194us) log.cc:844] Serialized 1650 byte log entry
1214 20:52:20.031392 (+ 7104us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.031528 (+ 136us) write_op.cc:301] APPLY: starting
1214 20:52:20.031544 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.031604 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.031648 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.031737 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.031774 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
1214 20:52:20.031781 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:20.031864 (+ 83us) log.cc:844] Serialized 41 byte log entry
1214 20:52:20.031922 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.031944 (+ 22us) write_op.cc:454] Released schema lock
1214 20:52:20.031951 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":148,"child_traces":[["op",{"apply.queue_time_us":90,"apply.run_cpu_time_us":472,"apply.run_wall_time_us":472,"num_ops":4,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":747,"prepare.run_wall_time_us":1014,"raft.queue_time_us":235,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":7247}]]}
I20251214 20:52:21.303030 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2688515bd1441fbad7516f8343ca775. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.304085 1206 ts_tablet_manager.cc:1434] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:21.307850 1206 tablet_bootstrap.cc:492] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.308899 743 tablet_service.cc:1505] Processing CreateTablet for tablet 8db4364b5023438bb43538855007c95e (DEFAULT_TABLE table=default.Table_dc35e103390944548c36be9ba4900882 [id=694ef81722d948638bbec8847c69d75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.309303 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db4364b5023438bb43538855007c95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.310871 1206 tablet_bootstrap.cc:654] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.311333 1762 raft_consensus.cc:493] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.311501 1762 raft_consensus.cc:515] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.312251 1762 leader_election.cc:290] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:21.308449 1650 raft_consensus.cc:493] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.312654 1650 raft_consensus.cc:515] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.313200 1650 leader_election.cc:290] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:21.313349 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e31615d1277412eb8bd80451f2c4366" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:21.314316 848 leader_election.cc:343] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.314904 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b70579ddf96f4499b4e4f235e8f69d67" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:21.315902 848 leader_election.cc:343] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.316896 1982 ts_tablet_manager.cc:1929] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.317847 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 66) took 2293 ms. Trace:
I20251214 20:52:21.318060 1856 rpcz_store.cc:276] 1214 20:52:18.994962 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:18.997208 (+ 2246us) service_pool.cc:225] Handling call
1214 20:52:21.288665 (+2291457us) ts_tablet_manager.cc:1151] Deleting tablet 35a529702f924a5c9bf53c3c668c6094
1214 20:52:21.288681 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.288829 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2251713}
I20251214 20:52:21.318506 1977 raft_consensus.cc:2243] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.318907 1856 tablet_replica.cc:333] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.318921 1977 raft_consensus.cc:2272] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.319787 1982 log.cc:1199] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/c25832da5c1644329e18653000f3330e
I20251214 20:52:21.320348 1982 ts_tablet_manager.cc:1950] T c25832da5c1644329e18653000f3330e P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.320662 750 tablet_service.cc:1505] Processing CreateTablet for tablet 825baba561ee4a8ebd44f75d8c51706a (DEFAULT_TABLE table=default.Table_2718b41c3d5643439f3a0d867fbb8116 [id=7e4dc62b017e469c8e9d5eddc35db955]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.321117 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825baba561ee4a8ebd44f75d8c51706a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.323364 741 tablet_service.cc:1505] Processing CreateTablet for tablet 2f9def8e4fd449599820a63845397750 (DEFAULT_TABLE table=default.Table_99e702b2f3fc436db82c793284a1670c [id=a9ff75b0f47f4601b2d489c3e3d6fce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.323724 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9def8e4fd449599820a63845397750. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.327790 891 tablet_service.cc:1505] Processing CreateTablet for tablet f7f06eded6c340c58101fcc644ebcf50 (DEFAULT_TABLE table=default.Table_8163ed145f64447c8eb6fc0f8d0a1d4d [id=c4c61d0a0bdf4db8bd6011c151d6b87c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.328305 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f06eded6c340c58101fcc644ebcf50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.328437 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 79) took 1283 ms. Trace:
I20251214 20:52:21.328575 1982 rpcz_store.cc:276] 1214 20:52:20.044476 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.046006 (+ 1530us) service_pool.cc:225] Handling call
1214 20:52:21.272906 (+1226900us) ts_tablet_manager.cc:1151] Deleting tablet c25832da5c1644329e18653000f3330e
1214 20:52:21.272916 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.299415 (+ 26499us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.309774 (+ 10359us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.328295 (+ 18521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1226684}
I20251214 20:52:21.328698 738 tablet_service.cc:1505] Processing CreateTablet for tablet 054cfd587d19409fbdf7f4848bcaf3f9 (DEFAULT_TABLE table=default.Table_816a237b5485489c9dc522f872701944 [id=9e91a928c95441bd8f0f1b939f2a7529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.329084 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054cfd587d19409fbdf7f4848bcaf3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.331449 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet c25832da5c1644329e18653000f3330e (table default.Table_c5d8c1bdf3344ec5a0d409a1abf98b45 [id=5427e484a5a94314a13be3f387c06d64]) successfully deleted
I20251214 20:52:21.335196 1856 raft_consensus.cc:2243] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.335537 1977 ts_tablet_manager.cc:1916] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.335801 1856 raft_consensus.cc:2272] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.335904 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 06381efe75284e649fd6d1a9903cfa9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.336534 1856 ts_tablet_manager.cc:1916] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.336589 881 tablet_service.cc:1505] Processing CreateTablet for tablet 8db4364b5023438bb43538855007c95e (DEFAULT_TABLE table=default.Table_dc35e103390944548c36be9ba4900882 [id=694ef81722d948638bbec8847c69d75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.337131 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db4364b5023438bb43538855007c95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.339272 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d6e7306254044449f1127fb963ca5f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.339735 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.349061 881 tablet_service.cc:1505] Processing CreateTablet for tablet 825baba561ee4a8ebd44f75d8c51706a (DEFAULT_TABLE table=default.Table_2718b41c3d5643439f3a0d867fbb8116 [id=7e4dc62b017e469c8e9d5eddc35db955]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.349674 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825baba561ee4a8ebd44f75d8c51706a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.340395 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 76472ce9005b4bd2bbc51278816e8ae0 (DEFAULT_TABLE table=default.Table_e74a0c96be654f4c808071da1ae734a0 [id=4b3f6c8d8cce4f7a96b2c677df6469ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.351117 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76472ce9005b4bd2bbc51278816e8ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.353828 1206 tablet_bootstrap.cc:492] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.354081 1206 ts_tablet_manager.cc:1403] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.005s
I20251214 20:52:21.355108 1206 raft_consensus.cc:359] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.355469 1206 raft_consensus.cc:385] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.355588 1206 raft_consensus.cc:740] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.355504 880 tablet_service.cc:1505] Processing CreateTablet for tablet 054cfd587d19409fbdf7f4848bcaf3f9 (DEFAULT_TABLE table=default.Table_816a237b5485489c9dc522f872701944 [id=9e91a928c95441bd8f0f1b939f2a7529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.356032 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054cfd587d19409fbdf7f4848bcaf3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.355962 1206 consensus_queue.cc:260] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.361302 1206 ts_tablet_manager.cc:1434] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251214 20:52:21.362332 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 649f5bd67cbb4f61b7066269f0bf9266 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.365437 743 tablet_service.cc:1505] Processing CreateTablet for tablet d171301941fa43908e05984c3b016007 (DEFAULT_TABLE table=default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.370039 1206 tablet_bootstrap.cc:492] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.371351 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d171301941fa43908e05984c3b016007. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.371974 1206 tablet_bootstrap.cc:654] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.376411 738 tablet_service.cc:1505] Processing CreateTablet for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (DEFAULT_TABLE table=default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.376811 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dcd511a5dca4a2cbda81fbc255ce354. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.379089 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 4501fd5566a34e2a82009cc9ce6bdd37 (DEFAULT_TABLE table=default.Table_a941092a7b2944bfa37073939660b587 [id=14f12b293a274ebeaf5bfefdff5b78cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.379705 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4501fd5566a34e2a82009cc9ce6bdd37. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.378839 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b1a937150ce4ecfa210c8465d682496 with delete_type TABLET_DATA_DELETED (replaced by tablet f1eb4c3aa654480caaa69dbd5047365c) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:21.381363 891 tablet_service.cc:1505] Processing CreateTablet for tablet 2f9def8e4fd449599820a63845397750 (DEFAULT_TABLE table=default.Table_99e702b2f3fc436db82c793284a1670c [id=a9ff75b0f47f4601b2d489c3e3d6fce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.383153 1033 tablet_service.cc:1505] Processing CreateTablet for tablet d2688515bd1441fbad7516f8343ca775 (DEFAULT_TABLE table=default.Table_d2beb8dc1b054cada5e4c2ec9fabc8d4 [id=03997e2eab154ef18b2e3774b5984764]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.383829 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 825baba561ee4a8ebd44f75d8c51706a (DEFAULT_TABLE table=default.Table_2718b41c3d5643439f3a0d867fbb8116 [id=7e4dc62b017e469c8e9d5eddc35db955]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.383829 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 054cfd587d19409fbdf7f4848bcaf3f9 (DEFAULT_TABLE table=default.Table_816a237b5485489c9dc522f872701944 [id=9e91a928c95441bd8f0f1b939f2a7529]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.384291 1982 tablet_replica.cc:333] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.384303 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 8db4364b5023438bb43538855007c95e (DEFAULT_TABLE table=default.Table_dc35e103390944548c36be9ba4900882 [id=694ef81722d948638bbec8847c69d75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.384511 1982 raft_consensus.cc:2243] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.384665 1021 tablet_service.cc:1505] Processing CreateTablet for tablet f7f06eded6c340c58101fcc644ebcf50 (DEFAULT_TABLE table=default.Table_8163ed145f64447c8eb6fc0f8d0a1d4d [id=c4c61d0a0bdf4db8bd6011c151d6b87c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.384871 1203 tablet_bootstrap.cc:492] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:21.386346 1982 raft_consensus.cc:2272] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.387027 1982 ts_tablet_manager.cc:1916] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.386957 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ca7b705bd1754ce19fbd7cda96364054"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:21.388617 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca7b705bd1754ce19fbd7cda96364054 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:21.391222 1203 ts_tablet_manager.cc:1403] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.143s user 0.010s sys 0.024s
W20251214 20:52:21.392076 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ee7d709316b24b8eb223a838710c2979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7d709316b24b8eb223a838710c2979 already in progress: creating tablet
W20251214 20:52:21.392549 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8d9f74fb2c6244a6887e3c922262bbab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9f74fb2c6244a6887e3c922262bbab already in progress: creating tablet
I20251214 20:52:21.392892 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 60ae3db968f6412aa0c3952fafe65f88 (table default.Table_01dcfe83d7854b4e9b8d6f62a557a0fe [id=d180e3a95c0840589c0aeec13ef1a6ca]) successfully deleted
W20251214 20:52:21.393163 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b2e83bca8ed642b6a1b705ef70fde27f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2e83bca8ed642b6a1b705ef70fde27f already in progress: creating tablet
I20251214 20:52:21.393462 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 40a9a6192c744282ab5672cbe15fd299 (table default.Table_847906e708bd41e9b9507efc8f52347f [id=4afcd94192854e049c21f8e6852cd56f]) successfully deleted
I20251214 20:52:21.393780 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 4b86f4adaf4540f48a4f42bc45e259e5 (table default.Table_69c0719fe94a4c7594516a739f6a40ea [id=a5424dd4eee64d0686d5e4a8a5c35213]) successfully deleted
W20251214 20:52:21.394071 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 35a529702f924a5c9bf53c3c668c6094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a529702f924a5c9bf53c3c668c6094 already in progress: creating tablet
I20251214 20:52:21.395005 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2688515bd1441fbad7516f8343ca775. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.395287 1034 tablet_service.cc:1505] Processing CreateTablet for tablet d171301941fa43908e05984c3b016007 (DEFAULT_TABLE table=default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.395324 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054cfd587d19409fbdf7f4848bcaf3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.396729 1856 ts_tablet_manager.cc:1929] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:21.397641 1856 log.cc:1199] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/3c3962c33a294fa98f8bd485268aee71
I20251214 20:52:21.398785 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 2f9def8e4fd449599820a63845397750 (DEFAULT_TABLE table=default.Table_99e702b2f3fc436db82c793284a1670c [id=a9ff75b0f47f4601b2d489c3e3d6fce0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:21.399616 991 consensus_peers.cc:597] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0279992d1f2419ab1caae116a0391b2. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:21.400256 738 tablet_service.cc:1505] Processing CreateTablet for tablet 3bed2c326906486e99300e05033c9c0f (DEFAULT_TABLE table=default.Table_725af8309e494ad3a5a40a73d2f585fa [id=c7816357b0fc4bc9aceaf1ad8d6ee30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.401017 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bed2c326906486e99300e05033c9c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.401909 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a34eb8fa3e34427495fa5985b7ba81bf" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:21.402400 1203 raft_consensus.cc:359] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.402835 713 leader_election.cc:304] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:21.402849 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (DEFAULT_TABLE table=default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:21.403414 713 consensus_peers.cc:597] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba50e32f7e724868bb891045a939e362. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:21.403441 991 consensus_peers.cc:597] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25974e5db555447baf5799775d15cd14. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:21.403870 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 3bed2c326906486e99300e05033c9c0f (DEFAULT_TABLE table=default.Table_725af8309e494ad3a5a40a73d2f585fa [id=c7816357b0fc4bc9aceaf1ad8d6ee30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.404657 1964 raft_consensus.cc:2749] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:21.405154 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 825baba561ee4a8ebd44f75d8c51706a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.406463 1856 ts_tablet_manager.cc:1950] T 3c3962c33a294fa98f8bd485268aee71 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.407116 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b70579ddf96f4499b4e4f235e8f69d67" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:21.409160 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e31615d1277412eb8bd80451f2c4366" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:21.410234 1203 raft_consensus.cc:385] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.410629 1203 raft_consensus.cc:740] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.410656 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a10663bd306d40e7a6375d06d69be22d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:21.411620 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bed2c326906486e99300e05033c9c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.411567 1203 consensus_queue.cc:260] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.411882 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9def8e4fd449599820a63845397750. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.412165 1203 ts_tablet_manager.cc:1434] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.021s user 0.000s sys 0.003s
W20251214 20:52:21.409974 851 leader_election.cc:343] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.406226 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f9def8e4fd449599820a63845397750. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.413125 851 leader_election.cc:304] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:21.414027 1650 raft_consensus.cc:2749] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:21.415198 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 78) took 2414 ms. Trace:
I20251214 20:52:21.415403 1054 raft_consensus.cc:2468] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:21.416677 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db4364b5023438bb43538855007c95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.417654 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 566c6e4761d5486d964e276d151e32b0 (DEFAULT_TABLE table=default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.413120 895 tablet_service.cc:1505] Processing CreateTablet for tablet d171301941fa43908e05984c3b016007 (DEFAULT_TABLE table=default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.418048 1856 rpcz_store.cc:276] 1214 20:52:18.996628 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.001137 (+ 4509us) service_pool.cc:225] Handling call
1214 20:52:21.318506 (+2317369us) ts_tablet_manager.cc:1151] Deleting tablet 3c3962c33a294fa98f8bd485268aee71
1214 20:52:21.318520 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.377980 (+ 59460us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.385586 (+ 7606us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.411257 (+ 25671us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":509,"tablet-delete.queue_time_us":2280659}
I20251214 20:52:21.419482 1203 tablet_bootstrap.cc:492] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:21.420504 851 leader_election.cc:304] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:21.421809 885 tablet_service.cc:1505] Processing CreateTablet for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (DEFAULT_TABLE table=default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.422071 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 3c3962c33a294fa98f8bd485268aee71 (table default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]) successfully deleted
I20251214 20:52:21.421809 1650 raft_consensus.cc:2804] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:21.422529 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d171301941fa43908e05984c3b016007. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.423370 757 tablet_service.cc:1505] Processing CreateTablet for tablet 566c6e4761d5486d964e276d151e32b0 (DEFAULT_TABLE table=default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.423692 1856 tablet_replica.cc:333] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.424266 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dcd511a5dca4a2cbda81fbc255ce354. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.425812 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566c6e4761d5486d964e276d151e32b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.425868 1856 raft_consensus.cc:2243] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.426955 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f06eded6c340c58101fcc644ebcf50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.426932 1856 raft_consensus.cc:2272] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.426913 1027 tablet_service.cc:1505] Processing CreateTablet for tablet e59b80ee919b4e1393fe317ed0e3a3d2 (DEFAULT_TABLE table=default.Table_ba7f3997d83f41b3a3a7019d61267c0e [id=70ba2f59c22f4f32b680b46a3c999b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.424973 713 leader_election.cc:304] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:21.427878 1964 raft_consensus.cc:2749] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:21.428501 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566c6e4761d5486d964e276d151e32b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.431684 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59b80ee919b4e1393fe317ed0e3a3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.435720 1856 ts_tablet_manager.cc:1916] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.438189 1203 tablet_bootstrap.cc:654] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.439612 1982 ts_tablet_manager.cc:1929] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.439743 1982 log.cc:1199] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/8faa7fd7315f485c82aee38bcf4cd8fc
I20251214 20:52:21.440044 1982 ts_tablet_manager.cc:1950] T 8faa7fd7315f485c82aee38bcf4cd8fc P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.440831 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 80) took 1392 ms. Trace:
I20251214 20:52:21.440932 1982 rpcz_store.cc:276] 1214 20:52:20.048813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.049428 (+ 615us) service_pool.cc:225] Handling call
1214 20:52:21.328887 (+1279459us) ts_tablet_manager.cc:1151] Deleting tablet 8faa7fd7315f485c82aee38bcf4cd8fc
1214 20:52:21.384223 (+ 55336us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.390355 (+ 6132us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.439500 (+ 49145us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.440765 (+ 1265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1279263}
I20251214 20:52:21.441398 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 8faa7fd7315f485c82aee38bcf4cd8fc (table default.Table_1d198a3975c24b84aaeb82c266c746b8 [id=bc62e2586be140efbc2bfb5a3b5baaa2]) successfully deleted
I20251214 20:52:21.443027 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_b2ac4ca26b6640a287e7f3b6974c2363"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:21.443544 738 tablet_service.cc:1505] Processing CreateTablet for tablet e59b80ee919b4e1393fe317ed0e3a3d2 (DEFAULT_TABLE table=default.Table_ba7f3997d83f41b3a3a7019d61267c0e [id=70ba2f59c22f4f32b680b46a3c999b53]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:21.444096 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2ac4ca26b6640a287e7f3b6974c2363 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:21.446134 1982 tablet_replica.cc:333] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.446357 1982 raft_consensus.cc:2243] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.446557 1982 raft_consensus.cc:2272] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.447180 1982 ts_tablet_manager.cc:1916] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.447427 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dcd511a5dca4a2cbda81fbc255ce354. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.450027 1206 tablet_bootstrap.cc:492] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.450698 894 tablet_service.cc:1505] Processing CreateTablet for tablet 3bed2c326906486e99300e05033c9c0f (DEFAULT_TABLE table=default.Table_725af8309e494ad3a5a40a73d2f585fa [id=c7816357b0fc4bc9aceaf1ad8d6ee30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.451571 1650 raft_consensus.cc:493] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:21.451709 1650 raft_consensus.cc:3060] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.453459 890 tablet_service.cc:1505] Processing CreateTablet for tablet 566c6e4761d5486d964e276d151e32b0 (DEFAULT_TABLE table=default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.454023 883 tablet_service.cc:1505] Processing CreateTablet for tablet e59b80ee919b4e1393fe317ed0e3a3d2 (DEFAULT_TABLE table=default.Table_ba7f3997d83f41b3a3a7019d61267c0e [id=70ba2f59c22f4f32b680b46a3c999b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:21.454232 1206 ts_tablet_manager.cc:1403] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.014s
I20251214 20:52:21.455456 1667 consensus_queue.cc:1048] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251214 20:52:21.459934 1928 consensus_queue.cc:1048] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20251214 20:52:21.460121 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59b80ee919b4e1393fe317ed0e3a3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.460597 1206 raft_consensus.cc:359] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.461186 1206 raft_consensus.cc:385] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.461326 1206 raft_consensus.cc:740] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.463286 1206 consensus_queue.cc:260] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.464133 1206 ts_tablet_manager.cc:1434] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20251214 20:52:21.465085 1206 tablet_bootstrap.cc:492] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.467934 1206 tablet_bootstrap.cc:654] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:21.477178 989 consensus_peers.cc:597] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:21.487941 1982 ts_tablet_manager.cc:1929] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.488567 1982 log.cc:1199] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/6b4f582d8165439d86d9228c146dc41a
I20251214 20:52:21.489356 1982 ts_tablet_manager.cc:1950] T 6b4f582d8165439d86d9228c146dc41a P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.491222 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 82) took 1433 ms. Trace:
I20251214 20:52:21.493358 1982 rpcz_store.cc:276] 1214 20:52:20.057762 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.059939 (+ 2177us) service_pool.cc:225] Handling call
1214 20:52:21.441203 (+1381264us) ts_tablet_manager.cc:1151] Deleting tablet 6b4f582d8165439d86d9228c146dc41a
1214 20:52:21.441212 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.466355 (+ 25143us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.469205 (+ 2850us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.491186 (+ 21981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1381042}
I20251214 20:52:21.494635 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 6b4f582d8165439d86d9228c146dc41a (table default.Table_22df18c031ab4778ae90eb3580e9b8b7 [id=3f105cc6dd6944ceb9783a1ebe923d11]) successfully deleted
I20251214 20:52:21.495630 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 84) took 1434 ms. Trace:
I20251214 20:52:21.496184 1982 rpcz_store.cc:276] 1214 20:52:20.061195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.061678 (+ 483us) service_pool.cc:225] Handling call
1214 20:52:21.495053 (+1433375us) ts_tablet_manager.cc:1151] Deleting tablet a063083f54ad49d2a842fa080d44ed85
1214 20:52:21.495063 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.495591 (+ 528us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1433178}
W20251214 20:52:21.497356 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet a063083f54ad49d2a842fa080d44ed85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a063083f54ad49d2a842fa080d44ed85 already in progress: creating tablet
I20251214 20:52:21.497864 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 88) took 1436 ms. Trace:
I20251214 20:52:21.498793 1982 rpcz_store.cc:276] 1214 20:52:20.061641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.063549 (+ 1908us) service_pool.cc:225] Handling call
1214 20:52:21.497734 (+1434185us) ts_tablet_manager.cc:1151] Deleting tablet 47e813373fbd482298312ead4f037c01
1214 20:52:21.497744 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.497835 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1434005}
I20251214 20:52:21.499562 1982 tablet_replica.cc:333] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
W20251214 20:52:21.499840 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 47e813373fbd482298312ead4f037c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e813373fbd482298312ead4f037c01 already in progress: creating tablet
I20251214 20:52:21.500594 1982 raft_consensus.cc:2243] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.501330 1982 raft_consensus.cc:2272] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.502393 1982 ts_tablet_manager.cc:1916] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.521195 1650 raft_consensus.cc:515] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.526391 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b70579ddf96f4499b4e4f235e8f69d67" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:21.526700 1056 raft_consensus.cc:3060] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.528136 1856 ts_tablet_manager.cc:1929] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.528342 1856 log.cc:1199] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/a10663bd306d40e7a6375d06d69be22d
I20251214 20:52:21.528582 1982 ts_tablet_manager.cc:1929] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:21.528721 1982 log.cc:1199] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/2da66ec83e9140f7a605f0c86abff468
I20251214 20:52:21.528862 1856 ts_tablet_manager.cc:1950] T a10663bd306d40e7a6375d06d69be22d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.529371 1982 ts_tablet_manager.cc:1950] T 2da66ec83e9140f7a605f0c86abff468 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.529728 1056 raft_consensus.cc:2468] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:21.529814 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 85) took 2494 ms. Trace:
I20251214 20:52:21.530228 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 86) took 1468 ms. Trace:
I20251214 20:52:21.530347 1982 rpcz_store.cc:276] 1214 20:52:20.061433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.064268 (+ 2835us) service_pool.cc:225] Handling call
1214 20:52:21.499495 (+1435227us) ts_tablet_manager.cc:1151] Deleting tablet 2da66ec83e9140f7a605f0c86abff468
1214 20:52:21.499503 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.506686 (+ 7183us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.528451 (+ 21765us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.530166 (+ 1715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1433658}
I20251214 20:52:21.530905 851 leader_election.cc:304] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:21.530942 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b70579ddf96f4499b4e4f235e8f69d67" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:21.530982 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 2da66ec83e9140f7a605f0c86abff468 (table default.Table_2325a2c5e97f45ebac4c943380bb3525 [id=0c6d9651f90f479293852cb3fb1d320f]) successfully deleted
I20251214 20:52:21.531306 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 89) took 1467 ms. Trace:
I20251214 20:52:21.531351 1762 raft_consensus.cc:2804] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:21.531558 1982 rpcz_store.cc:276] 1214 20:52:20.063974 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.064771 (+ 797us) service_pool.cc:225] Handling call
1214 20:52:21.530696 (+1465925us) ts_tablet_manager.cc:1151] Deleting tablet 35a529702f924a5c9bf53c3c668c6094
1214 20:52:21.530709 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.531252 (+ 543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464732}
W20251214 20:52:21.532090 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 35a529702f924a5c9bf53c3c668c6094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a529702f924a5c9bf53c3c668c6094 already in progress: creating tablet
I20251214 20:52:21.532168 1762 raft_consensus.cc:697] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:21.532511 1982 tablet_replica.cc:333] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.533041 1982 raft_consensus.cc:2243] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
W20251214 20:52:21.533344 848 leader_election.cc:343] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.533653 1762 consensus_queue.cc:237] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.534139 1982 raft_consensus.cc:2272] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.532979 1650 leader_election.cc:290] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:21.534849 1982 ts_tablet_manager.cc:1916] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.529933 1856 rpcz_store.cc:276] 1214 20:52:19.034832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.034919 (+ 87us) service_pool.cc:225] Handling call
1214 20:52:21.418382 (+2383463us) ts_tablet_manager.cc:1151] Deleting tablet a10663bd306d40e7a6375d06d69be22d
1214 20:52:21.418391 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.502089 (+ 83698us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.527961 (+ 25872us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.529739 (+ 1778us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2380138}
I20251214 20:52:21.536441 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet a10663bd306d40e7a6375d06d69be22d (table default.Table_f93eb4d354254ec79674c615721834b7 [id=e8e0419a222b460ab608def1752925f0]) successfully deleted
I20251214 20:52:21.539582 1856 tablet_replica.cc:333] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.539814 1856 raft_consensus.cc:2243] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.540180 1856 pending_rounds.cc:70] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:21.540261 1856 pending_rounds.cc:77] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493731323887616 op_type: NO_OP noop_request { }
I20251214 20:52:21.541522 1856 raft_consensus.cc:2889] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:21.541688 1856 raft_consensus.cc:2272] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.542348 1856 ts_tablet_manager.cc:1916] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.544402 1982 ts_tablet_manager.cc:1929] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.544708 1982 log.cc:1199] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/2842f1dc0c374d24946f90dac1cf8aa5
I20251214 20:52:21.545229 1982 ts_tablet_manager.cc:1950] T 2842f1dc0c374d24946f90dac1cf8aa5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.546118 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 59) took 1545 ms. Trace:
I20251214 20:52:21.546301 1982 rpcz_store.cc:276] 1214 20:52:20.001008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.064520 (+ 63512us) service_pool.cc:225] Handling call
1214 20:52:21.532418 (+1467898us) ts_tablet_manager.cc:1151] Deleting tablet 2842f1dc0c374d24946f90dac1cf8aa5
1214 20:52:21.532428 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.539516 (+ 7088us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.544263 (+ 4747us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.546062 (+ 1799us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1466378}
I20251214 20:52:21.546975 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 2842f1dc0c374d24946f90dac1cf8aa5 (table default.Table_a2a64eab2d9a493da6fee1051eadd37d [id=d929401b9c4c4b4ba21353fb8a8ecc3e]) successfully deleted
I20251214 20:52:21.547415 1202 tablet_bootstrap.cc:492] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:21.547652 1202 ts_tablet_manager.cc:1403] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.251s user 0.006s sys 0.026s
I20251214 20:52:21.548233 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 90) took 1483 ms. Trace:
I20251214 20:52:21.548710 1982 rpcz_store.cc:276] 1214 20:52:20.065046 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.065211 (+ 165us) service_pool.cc:225] Handling call
1214 20:52:21.548043 (+1482832us) ts_tablet_manager.cc:1151] Deleting tablet 5da9deeedce0421c9aeaebd71ba11b1b
1214 20:52:21.548056 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.548195 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1481950}
I20251214 20:52:21.549551 1982 tablet_replica.cc:333] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.549510 1202 raft_consensus.cc:359] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.549755 1202 raft_consensus.cc:385] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.549846 1202 raft_consensus.cc:740] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.549914 1982 raft_consensus.cc:2243] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.550119 1982 raft_consensus.cc:2272] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.550077 1202 consensus_queue.cc:260] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.550686 1202 ts_tablet_manager.cc:1434] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:21.551051 1202 tablet_bootstrap.cc:492] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:21.551153 1982 ts_tablet_manager.cc:1916] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.557006 1964 raft_consensus.cc:493] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.557206 1964 raft_consensus.cc:515] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.558022 1964 leader_election.cc:290] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.558970 1206 tablet_bootstrap.cc:492] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.559278 1206 ts_tablet_manager.cc:1403] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.095s user 0.008s sys 0.010s
I20251214 20:52:21.559422 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4b22043d81b493fb12116422b1d3fb1" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:21.560168 1206 raft_consensus.cc:359] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.560699 1856 ts_tablet_manager.cc:1929] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.560837 1206 raft_consensus.cc:385] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.560865 1856 log.cc:1199] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/a34eb8fa3e34427495fa5985b7ba81bf
I20251214 20:52:21.561034 1206 raft_consensus.cc:740] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.561254 1856 ts_tablet_manager.cc:1950] T a34eb8fa3e34427495fa5985b7ba81bf P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.561429 1206 consensus_queue.cc:260] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:21.561810 713 leader_election.cc:343] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: c4b22043d81b493fb12116422b1d3fb1
I20251214 20:52:21.561957 713 leader_election.cc:304] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:21.562119 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 79) took 2565 ms. Trace:
I20251214 20:52:21.562242 1964 raft_consensus.cc:2749] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:21.562216 1856 rpcz_store.cc:276] 1214 20:52:18.996699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.001443 (+ 4744us) service_pool.cc:225] Handling call
1214 20:52:21.539467 (+2538024us) ts_tablet_manager.cc:1151] Deleting tablet a34eb8fa3e34427495fa5985b7ba81bf
1214 20:52:21.539482 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.545525 (+ 6043us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.560571 (+ 15046us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.562065 (+ 1494us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2500930}
I20251214 20:52:21.562687 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet a34eb8fa3e34427495fa5985b7ba81bf (table default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]) successfully deleted
I20251214 20:52:21.563313 1206 ts_tablet_manager.cc:1434] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251214 20:52:21.563596 1856 tablet_replica.cc:333] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.563782 1856 raft_consensus.cc:2243] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.563956 1856 raft_consensus.cc:2272] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.564399 1206 tablet_bootstrap.cc:492] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.566532 1206 tablet_bootstrap.cc:654] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.571209 1856 ts_tablet_manager.cc:1916] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.574854 1982 ts_tablet_manager.cc:1929] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.575057 1982 log.cc:1199] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/63497e77baeb4d328be2421b5732a28a
I20251214 20:52:21.575742 1982 ts_tablet_manager.cc:1950] T 63497e77baeb4d328be2421b5732a28a P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.579775 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 85) took 1518 ms. Trace:
I20251214 20:52:21.581218 1982 rpcz_store.cc:276] 1214 20:52:20.061306 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.065539 (+ 4233us) service_pool.cc:225] Handling call
1214 20:52:21.549468 (+1483929us) ts_tablet_manager.cc:1151] Deleting tablet 63497e77baeb4d328be2421b5732a28a
1214 20:52:21.549479 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.554861 (+ 5382us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.574735 (+ 19874us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.579710 (+ 4975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1483065}
I20251214 20:52:21.582111 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 63497e77baeb4d328be2421b5732a28a (table default.Table_d553ca568da242899d768f9848baba3d [id=376e4eabd04d4a53ab2772323b8c41b7]) successfully deleted
I20251214 20:52:21.582434 1928 raft_consensus.cc:493] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.582665 1928 raft_consensus.cc:515] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.583325 1928 leader_election.cc:290] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.583863 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d171301941fa43908e05984c3b016007. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.583951 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb698ff30fcc47d09000cc68d2c5225b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:21.585069 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb698ff30fcc47d09000cc68d2c5225b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:21.585304 909 raft_consensus.cc:2468] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
W20251214 20:52:21.585772 989 leader_election.cc:343] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.585973 991 leader_election.cc:304] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:21.586385 1928 raft_consensus.cc:2804] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:21.586472 1928 raft_consensus.cc:493] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:21.586568 1928 raft_consensus.cc:3060] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.586953 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 92) took 1520 ms. Trace:
I20251214 20:52:21.587849 1982 rpcz_store.cc:276] 1214 20:52:20.066602 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.066810 (+ 208us) service_pool.cc:225] Handling call
1214 20:52:21.586789 (+1519979us) ts_tablet_manager.cc:1151] Deleting tablet ca263a80702f4f64aa7c872c2be08207
1214 20:52:21.586799 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.586906 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519739}
I20251214 20:52:21.587739 1206 tablet_bootstrap.cc:492] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.588241 1206 ts_tablet_manager.cc:1403] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
I20251214 20:52:21.588444 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bed2c326906486e99300e05033c9c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.589105 1206 raft_consensus.cc:359] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.589198 1928 raft_consensus.cc:515] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.589376 1206 raft_consensus.cc:385] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.589502 1206 raft_consensus.cc:740] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.589780 1206 consensus_queue.cc:260] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.590113 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb698ff30fcc47d09000cc68d2c5225b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:21.590313 774 raft_consensus.cc:3060] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:21.590629 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ca263a80702f4f64aa7c872c2be08207 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca263a80702f4f64aa7c872c2be08207 already in progress: creating tablet
I20251214 20:52:21.591511 1202 tablet_bootstrap.cc:654] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.592015 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566c6e4761d5486d964e276d151e32b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.592823 1928 leader_election.cc:290] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.597276 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59b80ee919b4e1393fe317ed0e3a3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:21.601255 1856 ts_tablet_manager.cc:1929] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:21.601774 1856 log.cc:1199] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/24ff58f321124d12b5330dca185d426e
I20251214 20:52:21.602046 774 raft_consensus.cc:2468] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:21.602349 1856 ts_tablet_manager.cc:1950] T 24ff58f321124d12b5330dca185d426e P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.602732 989 leader_election.cc:304] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:21.603173 1928 raft_consensus.cc:2804] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:21.603411 1928 raft_consensus.cc:697] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:21.603791 1928 consensus_queue.cc:237] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.604760 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb698ff30fcc47d09000cc68d2c5225b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:21.605039 912 raft_consensus.cc:3060] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.605050 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 86) took 2569 ms. Trace:
I20251214 20:52:21.605157 1856 rpcz_store.cc:276] 1214 20:52:19.035344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.035405 (+ 61us) service_pool.cc:225] Handling call
1214 20:52:21.563503 (+2528098us) ts_tablet_manager.cc:1151] Deleting tablet 24ff58f321124d12b5330dca185d426e
1214 20:52:21.563512 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.574240 (+ 10728us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.595117 (+ 20877us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.605004 (+ 9887us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2524907}
I20251214 20:52:21.606081 1206 ts_tablet_manager.cc:1434] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251214 20:52:21.606839 1206 tablet_bootstrap.cc:492] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.607225 1202 tablet_bootstrap.cc:492] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:21.607497 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 24ff58f321124d12b5330dca185d426e (table default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]) successfully deleted
I20251214 20:52:21.607571 1856 tablet_replica.cc:333] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.607760 1856 raft_consensus.cc:2243] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.609560 1856 raft_consensus.cc:2272] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.609714 1206 tablet_bootstrap.cc:654] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.610837 768 raft_consensus.cc:3060] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.614042 1928 consensus_queue.cc:1048] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:21.617726 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 93) took 1551 ms. Trace:
I20251214 20:52:21.617919 1982 rpcz_store.cc:276] 1214 20:52:20.066693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.067074 (+ 381us) service_pool.cc:225] Handling call
1214 20:52:21.591022 (+1523948us) ts_tablet_manager.cc:1151] Deleting tablet 292a28b12df246a8bc855d81c530746e
1214 20:52:21.591030 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.617668 (+ 26638us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1520146}
W20251214 20:52:21.618557 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 292a28b12df246a8bc855d81c530746e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292a28b12df246a8bc855d81c530746e already in progress: creating tablet
I20251214 20:52:21.618642 1856 ts_tablet_manager.cc:1916] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.619084 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 91) took 1553 ms. Trace:
I20251214 20:52:21.619239 1982 rpcz_store.cc:276] 1214 20:52:20.066482 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.070360 (+ 3878us) service_pool.cc:225] Handling call
1214 20:52:21.618929 (+1548569us) ts_tablet_manager.cc:1151] Deleting tablet 3d949abef1cb41a9905482d81a98d2c1
1214 20:52:21.618939 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.619034 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1547011}
W20251214 20:52:21.619732 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3d949abef1cb41a9905482d81a98d2c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d949abef1cb41a9905482d81a98d2c1 already in progress: creating tablet
I20251214 20:52:21.620105 1982 tablet_replica.cc:333] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.620440 1982 raft_consensus.cc:2243] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.620563 1202 ts_tablet_manager.cc:1403] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.011s
I20251214 20:52:21.620707 1982 raft_consensus.cc:2272] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.621232 1982 ts_tablet_manager.cc:1916] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.621995 1202 raft_consensus.cc:359] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.622290 1202 raft_consensus.cc:385] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.622377 1202 raft_consensus.cc:740] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.623018 1202 consensus_queue.cc:260] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.623667 1202 ts_tablet_manager.cc:1434] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:21.629547 1202 tablet_bootstrap.cc:492] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:21.631631 1202 tablet_bootstrap.cc:654] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.641992 912 raft_consensus.cc:2468] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
W20251214 20:52:21.644479 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:35099, user_credentials={real_user=slave}} blocked reactor thread for 205139us
I20251214 20:52:21.646992 1206 tablet_bootstrap.cc:492] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.647347 1206 ts_tablet_manager.cc:1403] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20251214 20:52:21.648295 1206 raft_consensus.cc:359] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.648607 1206 raft_consensus.cc:385] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.648741 1206 raft_consensus.cc:740] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.649080 1203 tablet_bootstrap.cc:492] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:21.649284 1203 ts_tablet_manager.cc:1403] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.237s user 0.018s sys 0.035s
I20251214 20:52:21.649688 1202 tablet_bootstrap.cc:492] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:21.649899 1202 ts_tablet_manager.cc:1403] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20251214 20:52:21.650271 1203 raft_consensus.cc:359] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.650470 1203 raft_consensus.cc:385] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.650579 1203 raft_consensus.cc:740] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.650856 1203 consensus_queue.cc:260] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.652076 1203 ts_tablet_manager.cc:1434] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:21.654489 1203 tablet_bootstrap.cc:492] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:21.658002 1203 tablet_bootstrap.cc:654] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.660584 1856 ts_tablet_manager.cc:1929] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.649210 1206 consensus_queue.cc:260] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.663966 1856 log.cc:1199] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/b1df0dad2fb040a5a12e0401e00c87db
I20251214 20:52:21.664908 1856 ts_tablet_manager.cc:1950] T b1df0dad2fb040a5a12e0401e00c87db P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.669740 1202 raft_consensus.cc:359] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.670063 1202 raft_consensus.cc:385] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.670055 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 82) took 2673 ms. Trace:
I20251214 20:52:21.670190 1202 raft_consensus.cc:740] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.670168 1856 rpcz_store.cc:276] 1214 20:52:19.011658 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.027680 (+ 16022us) service_pool.cc:225] Handling call
1214 20:52:21.607498 (+2579818us) ts_tablet_manager.cc:1151] Deleting tablet b1df0dad2fb040a5a12e0401e00c87db
1214 20:52:21.607508 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.621797 (+ 14289us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.660408 (+ 38611us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.669992 (+ 9584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2567316}
I20251214 20:52:21.670473 1202 consensus_queue.cc:260] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.671006 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet b1df0dad2fb040a5a12e0401e00c87db (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]) successfully deleted
I20251214 20:52:21.671063 1202 ts_tablet_manager.cc:1434] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251214 20:52:21.671834 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_7923a9cba4ce4b27a12617e394ffbf2e" } modify_external_catalogs: true
I20251214 20:52:21.672124 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_7923a9cba4ce4b27a12617e394ffbf2e" } modify_external_catalogs: true
I20251214 20:52:21.673229 1856 tablet_replica.cc:333] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.674135 1202 tablet_bootstrap.cc:492] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:21.675707 1206 ts_tablet_manager.cc:1434] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251214 20:52:21.675102 1856 raft_consensus.cc:2243] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.676101 774 raft_consensus.cc:3060] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.676924 1856 raft_consensus.cc:2272] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.677533 1856 ts_tablet_manager.cc:1916] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.681685 2079 consensus_queue.cc:1048] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:21.685634 712 consensus_peers.cc:597] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6c0f445cd7e44769d79ad5566fd8d97. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:21.691588 1667 consensus_queue.cc:1048] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:21.699205 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:21.699563 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d01db63fd5d2474a9defdce0b98fc29d (table default.Table_13f016135de44cf78187fbc4f9f8c622 [id=fc160acc971e4f7d8290560ee8aab065]). Expected version 1 got 0
I20251214 20:52:21.699872 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:21.700117 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_76f578444d71410a93c3f78ff50b7951 [id=6157817e96d642d09f9941ad7f7e82ee]). Expected version 2 got 0
I20251214 20:52:21.700358 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:21.701870 1977 ts_tablet_manager.cc:1929] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.702091 1977 log.cc:1199] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/a10663bd306d40e7a6375d06d69be22d
I20251214 20:52:21.702595 1977 ts_tablet_manager.cc:1950] T a10663bd306d40e7a6375d06d69be22d P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:21.703888 1206 tablet_bootstrap.cc:492] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.704008 1977 tablet_replica.cc:333] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:21.704272 1977 raft_consensus.cc:2243] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.705582 1203 tablet_bootstrap.cc:492] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:21.705825 1203 ts_tablet_manager.cc:1403] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.004s
I20251214 20:52:21.706241 1206 tablet_bootstrap.cc:654] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.706689 1203 raft_consensus.cc:359] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.706970 1203 raft_consensus.cc:385] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.707078 1203 raft_consensus.cc:740] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.712172 1964 consensus_queue.cc:1048] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:21.718276 1977 pending_rounds.cc:70] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:21.703724 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet a10663bd306d40e7a6375d06d69be22d (table default.Table_f93eb4d354254ec79674c615721834b7 [id=e8e0419a222b460ab608def1752925f0]) successfully deleted
I20251214 20:52:21.719199 1977 pending_rounds.cc:77] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493723268583424 op_type: NO_OP noop_request { }
I20251214 20:52:21.719292 1982 ts_tablet_manager.cc:1929] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251214 20:52:21.723846 1982 log.cc:1199] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/2b48bb260e854beb9ec61ce9c79b46c2
W20251214 20:52:21.724072 991 consensus_peers.cc:597] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b48bb260e854beb9ec61ce9c79b46c2. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:21.725248 2079 consensus_queue.cc:1048] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 20:52:21.726236 1203 consensus_queue.cc:260] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.726980 1203 ts_tablet_manager.cc:1434] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20251214 20:52:21.727893 1203 tablet_bootstrap.cc:492] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:21.729414 1982 ts_tablet_manager.cc:1950] T 2b48bb260e854beb9ec61ce9c79b46c2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.719097 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8e11723c042a4637839f1b13bed0935a (table default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]). Expected version 1 got 0
I20251214 20:52:21.728156 1202 tablet_bootstrap.cc:654] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.731323 1206 tablet_bootstrap.cc:492] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.731554 1206 ts_tablet_manager.cc:1403] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.001s
I20251214 20:52:21.732394 1856 ts_tablet_manager.cc:1929] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.732400 1206 raft_consensus.cc:359] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.732749 1206 raft_consensus.cc:385] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.732882 1206 raft_consensus.cc:740] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.733155 1977 raft_consensus.cc:2889] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:21.733563 1977 raft_consensus.cc:2272] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.734342 1977 ts_tablet_manager.cc:1916] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.734521 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:21.735321 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet afd51f330eb649b998a22a343d993b30 (table default.Table_9151bdce51684c39901786aef1c63ec5 [id=f4157d7b9fc848a8b7fa08742b456167]). Expected version 1 got 0
I20251214 20:52:21.737356 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]). Expected version 1 got 0
I20251214 20:52:21.741391 1856 log.cc:1199] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/c16895ca08d94b56b5c4bc9352e330ef
I20251214 20:52:21.743378 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:21.744104 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:21.744822 1856 ts_tablet_manager.cc:1950] T c16895ca08d94b56b5c4bc9352e330ef P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.733708 1203 tablet_bootstrap.cc:654] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.747999 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:21.748219 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
I20251214 20:52:21.748389 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 3fa0317e999b483788e5c5652e6e83de (table default.Table_3c676f861c28408f99faac2ab8f7eec1 [id=ef22180b36a241fea13d0e622c02b50a]). Expected version 3 got 2
I20251214 20:52:21.748697 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:21.748986 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 1 got 0
I20251214 20:52:21.751365 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 81) took 2754 ms. Trace:
I20251214 20:52:21.751616 1856 rpcz_store.cc:276] 1214 20:52:18.996819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.024403 (+ 27584us) service_pool.cc:225] Handling call
1214 20:52:21.673112 (+2648709us) ts_tablet_manager.cc:1151] Deleting tablet c16895ca08d94b56b5c4bc9352e330ef
1214 20:52:21.673124 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.716328 (+ 43204us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.732232 (+ 15904us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.751303 (+ 19071us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2632501}
I20251214 20:52:21.752411 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 83) took 2725 ms. Trace:
I20251214 20:52:21.752969 1856 rpcz_store.cc:276] 1214 20:52:19.027047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.028275 (+ 1228us) service_pool.cc:225] Handling call
1214 20:52:21.752212 (+2723937us) ts_tablet_manager.cc:1151] Deleting tablet 012b81223d0f4ae9b742014e329f11a9
1214 20:52:21.752222 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.752375 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2711456}
I20251214 20:52:21.753484 1856 tablet_replica.cc:333] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.753798 1856 raft_consensus.cc:2243] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.754236 1856 raft_consensus.cc:2272] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.755350 1963 raft_consensus.cc:493] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.755956 1963 raft_consensus.cc:515] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.756500 1206 consensus_queue.cc:260] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.756743 1963 leader_election.cc:290] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.757278 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet c16895ca08d94b56b5c4bc9352e330ef (table default.Table_9f6bb794724346069627cd2f14a5cd1b [id=4e30619d28324239bc069a5ce053b167]) successfully deleted
W20251214 20:52:21.757611 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 012b81223d0f4ae9b742014e329f11a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012b81223d0f4ae9b742014e329f11a9 already in progress: creating tablet
I20251214 20:52:21.757566 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a8662a634a4befbd2d6251000d2059" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:21.763756 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:21.764135 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 94) took 1687 ms. Trace:
I20251214 20:52:21.764343 1206 ts_tablet_manager.cc:1434] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20251214 20:52:21.764499 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:21.764451 1982 rpcz_store.cc:276] 1214 20:52:20.066776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.072313 (+ 5537us) service_pool.cc:225] Handling call
1214 20:52:21.620023 (+1547710us) ts_tablet_manager.cc:1151] Deleting tablet 2b48bb260e854beb9ec61ce9c79b46c2
1214 20:52:21.620032 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.701738 (+ 81706us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.719146 (+ 17408us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.754649 (+ 35503us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1545391}
I20251214 20:52:21.768182 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:21.758446 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07a8662a634a4befbd2d6251000d2059" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:21.768803 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]) successfully deleted
I20251214 20:52:21.769181 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:21.769503 713 leader_election.cc:304] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:21.770277 1964 raft_consensus.cc:2749] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:21.767251 2038 raft_consensus.cc:493] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:21.767357 989 consensus_peers.cc:597] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:21.771332 2038 raft_consensus.cc:515] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.772307 1977 ts_tablet_manager.cc:1929] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.772459 1977 log.cc:1199] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/52daf1204bf142e7b82852bdec6582fb
I20251214 20:52:21.772895 1977 ts_tablet_manager.cc:1950] T 52daf1204bf142e7b82852bdec6582fb P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:21.773309 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e16fe751f10f4616a00d0af811998729" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:21.773401 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 87) took 1711 ms. Trace:
I20251214 20:52:21.773501 1982 rpcz_store.cc:276] 1214 20:52:20.061545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.071994 (+ 10449us) service_pool.cc:225] Handling call
1214 20:52:21.768748 (+1696754us) ts_tablet_manager.cc:1151] Deleting tablet cd88d1e95ee04ef596863e1ae3687de0
1214 20:52:21.773267 (+ 4519us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.773365 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693977}
I20251214 20:52:21.773598 909 raft_consensus.cc:2468] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:21.773887 1977 tablet_replica.cc:333] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:21.773912 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e16fe751f10f4616a00d0af811998729" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:21.774093 1977 raft_consensus.cc:2243] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.774092 991 leader_election.cc:304] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:21.774258 1977 raft_consensus.cc:2272] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:21.774346 989 leader_election.cc:343] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.774468 2079 raft_consensus.cc:2804] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:21.774567 2079 raft_consensus.cc:493] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:21.774708 2079 raft_consensus.cc:3060] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.774755 1977 ts_tablet_manager.cc:1916] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.774886 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 95) took 1702 ms. Trace:
I20251214 20:52:21.775030 1982 rpcz_store.cc:276] 1214 20:52:20.072613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.073867 (+ 1254us) service_pool.cc:225] Handling call
1214 20:52:21.774758 (+1700891us) ts_tablet_manager.cc:1151] Deleting tablet 242a77ae57174de48f4eaf43384ee0ba
1214 20:52:21.774766 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.774859 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1699909}
W20251214 20:52:21.775758 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet cd88d1e95ee04ef596863e1ae3687de0 because the tablet was not found. No further retry: Not found: Tablet not found: cd88d1e95ee04ef596863e1ae3687de0
W20251214 20:52:21.776149 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 242a77ae57174de48f4eaf43384ee0ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 242a77ae57174de48f4eaf43384ee0ba already in progress: creating tablet
I20251214 20:52:21.776463 1982 tablet_replica.cc:333] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.776608 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 52daf1204bf142e7b82852bdec6582fb (table default.Table_5fbbdeeea9a14e17bf6f9d0cc4a56d98 [id=a0e13ae34009499d91b4461520c292c1]) successfully deleted
I20251214 20:52:21.776976 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:21.777316 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:21.777721 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 02e256d4537249a394b80dc36beefef4 (table default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]). Expected version 1 got 0
I20251214 20:52:21.778141 1762 raft_consensus.cc:493] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.778389 1762 raft_consensus.cc:515] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.778659 2079 raft_consensus.cc:515] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.779209 1762 leader_election.cc:290] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:21.779981 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7d709316b24b8eb223a838710c2979" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:21.780182 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e16fe751f10f4616a00d0af811998729" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:21.780360 909 raft_consensus.cc:3060] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:21.780577 848 leader_election.cc:343] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.781354 2079 leader_election.cc:290] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.781378 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7d709316b24b8eb223a838710c2979" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:21.782114 851 leader_election.cc:343] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.782225 851 leader_election.cc:304] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:21.782675 1856 ts_tablet_manager.cc:1916] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.782677 1762 raft_consensus.cc:2749] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:21.787333 1206 tablet_bootstrap.cc:492] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.787405 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e16fe751f10f4616a00d0af811998729" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:21.787830 1982 raft_consensus.cc:2243] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.788694 1982 raft_consensus.cc:2272] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.788731 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
W20251214 20:52:21.788941 989 leader_election.cc:343] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.789162 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:21.789350 1982 ts_tablet_manager.cc:1916] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.789808 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet feb19ea389af4ea6bb158a0e8aaeb094 (table default.Table_7660878c81e04c17b9b6fa8d61ef43a6 [id=59d9716c5cee42c391018111323c5181]). Expected version 1 got 0
I20251214 20:52:21.790006 1203 tablet_bootstrap.cc:492] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:21.790163 1203 ts_tablet_manager.cc:1403] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.001s
I20251214 20:52:21.790210 909 raft_consensus.cc:2468] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:21.790580 455 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 727e7b61ae3942098712720d344c5310 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]). Expected version 1 got 0
I20251214 20:52:21.790834 991 leader_election.cc:304] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:21.781579 2038 leader_election.cc:290] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.792250 2079 raft_consensus.cc:2804] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:21.792440 2079 raft_consensus.cc:697] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:21.792780 2079 consensus_queue.cc:237] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.791989 1203 raft_consensus.cc:359] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.794090 1203 raft_consensus.cc:385] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.794231 1203 raft_consensus.cc:740] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.794523 1203 consensus_queue.cc:260] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.795233 1203 ts_tablet_manager.cc:1434] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251214 20:52:21.795670 1203 tablet_bootstrap.cc:492] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:21.798512 1203 tablet_bootstrap.cc:654] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.807933 1202 tablet_bootstrap.cc:492] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:21.808151 1202 ts_tablet_manager.cc:1403] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.134s user 0.005s sys 0.007s
I20251214 20:52:21.808981 1202 raft_consensus.cc:359] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.809188 1202 raft_consensus.cc:385] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.809252 1202 raft_consensus.cc:740] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.809470 1202 consensus_queue.cc:260] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.810719 1202 ts_tablet_manager.cc:1434] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:21.811970 1202 tablet_bootstrap.cc:492] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:21.813782 1202 tablet_bootstrap.cc:654] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.817883 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:21.818830 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
I20251214 20:52:21.822548 1977 ts_tablet_manager.cc:1929] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:21.822687 1977 log.cc:1199] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/7fcfcfe81fe24d369eea696efc61d21c
I20251214 20:52:21.823037 1977 ts_tablet_manager.cc:1950] T 7fcfcfe81fe24d369eea696efc61d21c P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:21.823485 1206 tablet_bootstrap.cc:654] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.824170 1202 tablet_bootstrap.cc:492] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:21.824357 1202 ts_tablet_manager.cc:1403] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20251214 20:52:21.825162 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:21.825490 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 727e7b61ae3942098712720d344c5310 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]). Expected version 1 got 0
I20251214 20:52:21.825450 1202 raft_consensus.cc:359] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.825686 1202 raft_consensus.cc:385] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.825781 1202 raft_consensus.cc:740] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.826066 1202 consensus_queue.cc:260] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.826771 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:21.827391 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 7fcfcfe81fe24d369eea696efc61d21c (table default.Table_caa5b0dabd9d4ea8a8b16c550407dc53 [id=4c909a51df124835bd806b8c6e53214e]) successfully deleted
I20251214 20:52:21.827855 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:21.828024 1202 ts_tablet_manager.cc:1434] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20251214 20:52:21.828194 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 24ff58f321124d12b5330dca185d426e because the tablet was not found. No further retry: Not found: Tablet not found: 24ff58f321124d12b5330dca185d426e
I20251214 20:52:21.828280 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 3fa0317e999b483788e5c5652e6e83de (table default.Table_3c676f861c28408f99faac2ab8f7eec1 [id=ef22180b36a241fea13d0e622c02b50a]). Expected version 3 got 2
I20251214 20:52:21.828963 1977 tablet_replica.cc:333] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:21.829267 1977 raft_consensus.cc:2243] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.829591 1977 raft_consensus.cc:2272] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.830150 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:21.830440 1202 tablet_bootstrap.cc:492] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:21.830915 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d01db63fd5d2474a9defdce0b98fc29d (table default.Table_13f016135de44cf78187fbc4f9f8c622 [id=fc160acc971e4f7d8290560ee8aab065]). Expected version 1 got 0
I20251214 20:52:21.832481 1982 ts_tablet_manager.cc:1929] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:21.833026 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 1 got 0
I20251214 20:52:21.833371 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:21.833560 1202 tablet_bootstrap.cc:654] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.833793 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:21.837652 1977 ts_tablet_manager.cc:1916] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.837994 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:21.838240 1982 log.cc:1199] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/c4b22043d81b493fb12116422b1d3fb1
I20251214 20:52:21.838485 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 8e11723c042a4637839f1b13bed0935a (table default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]). Expected version 1 got 0
I20251214 20:52:21.838976 1982 ts_tablet_manager.cc:1950] T c4b22043d81b493fb12116422b1d3fb1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.839303 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet feb19ea389af4ea6bb158a0e8aaeb094 (table default.Table_7660878c81e04c17b9b6fa8d61ef43a6 [id=59d9716c5cee42c391018111323c5181]). Expected version 1 got 0
I20251214 20:52:21.841672 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 81) took 1790 ms. Trace:
I20251214 20:52:21.839861 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:21.842110 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:21.843020 2038 raft_consensus.cc:493] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.843144 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:21.843288 2038 raft_consensus.cc:515] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.843498 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 02e256d4537249a394b80dc36beefef4 (table default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]). Expected version 1 got 0
I20251214 20:52:21.843698 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:21.844007 1982 rpcz_store.cc:276] 1214 20:52:20.051567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.074885 (+ 23318us) service_pool.cc:225] Handling call
1214 20:52:21.776407 (+1701522us) ts_tablet_manager.cc:1151] Deleting tablet c4b22043d81b493fb12116422b1d3fb1
1214 20:52:21.776416 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.821173 (+ 44757us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.832344 (+ 11171us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.841610 (+ 9266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1700984}
I20251214 20:52:21.844195 2038 leader_election.cc:290] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.844641 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet c4b22043d81b493fb12116422b1d3fb1 (table default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]) successfully deleted
I20251214 20:52:21.844588 1982 tablet_replica.cc:333] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.844956 1982 raft_consensus.cc:2243] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.845237 1982 raft_consensus.cc:2272] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.845477 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e256d4537249a394b80dc36beefef4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:21.840399 1206 tablet_bootstrap.cc:492] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.845718 770 raft_consensus.cc:2468] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:21.846235 989 leader_election.cc:304] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:21.846613 2038 raft_consensus.cc:2804] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:21.846719 2038 raft_consensus.cc:493] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:21.846841 2038 raft_consensus.cc:3060] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.846933 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e256d4537249a394b80dc36beefef4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:21.847165 909 raft_consensus.cc:2468] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:21.847189 1982 ts_tablet_manager.cc:1916] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.849951 2038 raft_consensus.cc:515] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.851457 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e256d4537249a394b80dc36beefef4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:21.851549 2038 leader_election.cc:290] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.851640 770 raft_consensus.cc:3060] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.851789 1206 ts_tablet_manager.cc:1403] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.084s user 0.007s sys 0.000s
I20251214 20:52:21.852139 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e256d4537249a394b80dc36beefef4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:21.852334 909 raft_consensus.cc:3060] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.853165 1206 raft_consensus.cc:359] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.853446 1206 raft_consensus.cc:385] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.853523 1206 raft_consensus.cc:740] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.853983 1856 ts_tablet_manager.cc:1929] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.854135 1856 log.cc:1199] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/a0279992d1f2419ab1caae116a0391b2
I20251214 20:52:21.854704 1856 ts_tablet_manager.cc:1950] T a0279992d1f2419ab1caae116a0391b2 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.855430 909 raft_consensus.cc:2468] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:21.855473 770 raft_consensus.cc:2468] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:21.846623 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:21.856102 989 leader_election.cc:304] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:21.856536 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:21.856593 2038 raft_consensus.cc:2804] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:21.856899 2038 raft_consensus.cc:697] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:21.857213 2038 consensus_queue.cc:237] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.858376 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:21.859560 1203 tablet_bootstrap.cc:492] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:21.859783 1203 ts_tablet_manager.cc:1403] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.003s
I20251214 20:52:21.860038 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_76f578444d71410a93c3f78ff50b7951 [id=6157817e96d642d09f9941ad7f7e82ee]). Expected version 2 got 0
I20251214 20:52:21.860329 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:21.860689 1203 raft_consensus.cc:359] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.860924 1203 raft_consensus.cc:385] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.861006 1203 raft_consensus.cc:740] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.861392 1977 ts_tablet_manager.cc:1929] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.861545 1977 log.cc:1199] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/cd88d1e95ee04ef596863e1ae3687de0
I20251214 20:52:21.861531 1206 consensus_queue.cc:260] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.863512 1203 consensus_queue.cc:260] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.864223 1206 ts_tablet_manager.cc:1434] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251214 20:52:21.868062 1977 ts_tablet_manager.cc:1950] T cd88d1e95ee04ef596863e1ae3687de0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:21.868672 1206 tablet_bootstrap.cc:492] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.871644 1206 tablet_bootstrap.cc:654] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.873904 1202 tablet_bootstrap.cc:492] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:21.874173 1202 ts_tablet_manager.cc:1403] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.010s
I20251214 20:52:21.875083 1202 raft_consensus.cc:359] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.875442 1202 raft_consensus.cc:385] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.875671 1202 raft_consensus.cc:740] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.876001 1202 consensus_queue.cc:260] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.876775 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 100) took 1041 ms. Trace:
I20251214 20:52:21.876886 1977 rpcz_store.cc:276] 1214 20:52:20.835428 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.837797 (+ 2369us) service_pool.cc:225] Handling call
1214 20:52:21.828510 (+990713us) ts_tablet_manager.cc:1151] Deleting tablet cd88d1e95ee04ef596863e1ae3687de0
1214 20:52:21.828882 (+ 372us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.858079 (+ 29197us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.861256 (+ 3177us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.876707 (+ 15451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":971245}
I20251214 20:52:21.877132 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 87) took 2834 ms. Trace:
I20251214 20:52:21.877466 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet cd88d1e95ee04ef596863e1ae3687de0 (table default.Table_bda8a86e13ba427e94e8d85a318281ad [id=2018af6df0b448bd8996691cb5a1f62c]) successfully deleted
I20251214 20:52:21.877386 1856 rpcz_store.cc:276] 1214 20:52:19.039463 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.039518 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:21.753419 (+2713901us) ts_tablet_manager.cc:1151] Deleting tablet a0279992d1f2419ab1caae116a0391b2
1214 20:52:21.753429 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.786327 (+ 32898us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.853792 (+ 67465us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.872789 (+ 18997us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2703098}
I20251214 20:52:21.877820 1203 ts_tablet_manager.cc:1434] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20251214 20:52:21.877866 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet a0279992d1f2419ab1caae116a0391b2 (table default.Table_4e46507057c94aa7a347491bf2e482ff [id=3906806ef8294342a79b9face8753212]) successfully deleted
I20251214 20:52:21.878931 1977 tablet_replica.cc:333] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:21.880178 1977 raft_consensus.cc:2243] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.880373 1977 raft_consensus.cc:2272] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.880717 1203 tablet_bootstrap.cc:492] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:21.881640 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 88) took 2828 ms. Trace:
I20251214 20:52:21.881742 1856 rpcz_store.cc:276] 1214 20:52:19.053377 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.053461 (+ 84us) service_pool.cc:225] Handling call
1214 20:52:21.878618 (+2825157us) ts_tablet_manager.cc:1151] Deleting tablet bf8643c7e4094f529cd55656c7b3edb6
1214 20:52:21.878629 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.881538 (+ 2909us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2823567}
W20251214 20:52:21.883626 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet bf8643c7e4094f529cd55656c7b3edb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf8643c7e4094f529cd55656c7b3edb6 already in progress: creating tablet
I20251214 20:52:21.884249 1982 ts_tablet_manager.cc:1929] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.884387 1982 log.cc:1199] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/40a9a6192c744282ab5672cbe15fd299
I20251214 20:52:21.884850 1982 ts_tablet_manager.cc:1950] T 40a9a6192c744282ab5672cbe15fd299 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.885213 1206 tablet_bootstrap.cc:492] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.885450 1203 tablet_bootstrap.cc:654] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.886061 1977 ts_tablet_manager.cc:1916] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.887373 1206 ts_tablet_manager.cc:1403] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20251214 20:52:21.885658 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 97) took 1721 ms. Trace:
I20251214 20:52:21.888175 1982 rpcz_store.cc:276] 1214 20:52:20.164334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.164385 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:21.844500 (+1680115us) ts_tablet_manager.cc:1151] Deleting tablet 40a9a6192c744282ab5672cbe15fd299
1214 20:52:21.844511 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.850283 (+ 5772us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.884082 (+ 33799us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.885618 (+ 1536us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679753}
I20251214 20:52:21.890154 1202 ts_tablet_manager.cc:1434] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251214 20:52:21.890533 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet feb19ea389af4ea6bb158a0e8aaeb094 (table default.Table_7660878c81e04c17b9b6fa8d61ef43a6 [id=59d9716c5cee42c391018111323c5181]). Expected version 1 got 0
I20251214 20:52:21.891348 1982 tablet_replica.cc:333] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:21.891861 1982 raft_consensus.cc:2243] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.892558 1982 raft_consensus.cc:2272] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.893669 1982 ts_tablet_manager.cc:1916] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.894932 1202 tablet_bootstrap.cc:492] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:21.894402 1206 raft_consensus.cc:359] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.895293 1206 raft_consensus.cc:385] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.895884 1206 raft_consensus.cc:740] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.897110 1202 tablet_bootstrap.cc:654] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.897610 1206 consensus_queue.cc:260] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.899497 1206 ts_tablet_manager.cc:1434] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251214 20:52:21.900213 1206 tablet_bootstrap.cc:492] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.906911 1856 tablet_replica.cc:333] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:21.907848 1856 raft_consensus.cc:2243] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:21.909056 1856 raft_consensus.cc:2272] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.910071 1856 ts_tablet_manager.cc:1916] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.912828 2038 raft_consensus.cc:493] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.913038 2038 raft_consensus.cc:515] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.913141 1762 raft_consensus.cc:493] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.913339 1762 raft_consensus.cc:515] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.914027 1762 leader_election.cc:290] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:21.914273 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcdd6bafa5c4aa6b59668cde523d010" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:21.914438 2038 leader_election.cc:290] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.914494 770 raft_consensus.cc:2468] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:21.914515 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcdd6bafa5c4aa6b59668cde523d010" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:21.895756 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 40a9a6192c744282ab5672cbe15fd299 (table default.Table_847906e708bd41e9b9507efc8f52347f [id=4afcd94192854e049c21f8e6852cd56f]) successfully deleted
I20251214 20:52:21.914779 909 raft_consensus.cc:2468] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:21.891005 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:21.914950 989 leader_election.cc:304] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:21.915340 2038 raft_consensus.cc:2804] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:21.915352 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a063083f54ad49d2a842fa080d44ed85" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:21.915897 848 leader_election.cc:343] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.916430 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:21.916729 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:21.916970 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:21.917237 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 727e7b61ae3942098712720d344c5310 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]). Expected version 1 got 0
I20251214 20:52:21.916692 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a063083f54ad49d2a842fa080d44ed85" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:21.917776 1054 raft_consensus.cc:2468] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:21.918085 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:21.918272 851 leader_election.cc:304] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:21.918735 1762 raft_consensus.cc:2804] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:21.918867 1762 raft_consensus.cc:493] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:21.918963 1762 raft_consensus.cc:3060] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.919282 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:21.920579 2038 raft_consensus.cc:493] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:21.920699 2038 raft_consensus.cc:3060] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.921172 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:21.923498 2038 raft_consensus.cc:515] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.924062 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:21.927832 1762 raft_consensus.cc:515] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.928741 1203 tablet_bootstrap.cc:492] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:21.928922 1203 ts_tablet_manager.cc:1403] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.003s
I20251214 20:52:21.929307 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a063083f54ad49d2a842fa080d44ed85" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:21.929544 1054 raft_consensus.cc:3060] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.929848 1203 raft_consensus.cc:359] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.930025 1203 raft_consensus.cc:385] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.930135 1203 raft_consensus.cc:740] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.930356 1203 consensus_queue.cc:260] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.931061 1206 tablet_bootstrap.cc:654] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.932894 1203 ts_tablet_manager.cc:1434] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:21.932943 1054 raft_consensus.cc:2468] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:21.933233 1203 tablet_bootstrap.cc:492] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:21.934046 851 leader_election.cc:304] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:21.934399 1650 raft_consensus.cc:2804] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:21.934525 1650 raft_consensus.cc:697] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:21.934803 1650 consensus_queue.cc:237] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.936441 1964 consensus_queue.cc:1048] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:21.937996 1762 leader_election.cc:290] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:21.937935 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcdd6bafa5c4aa6b59668cde523d010" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:21.938149 770 raft_consensus.cc:3060] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.938915 2038 leader_election.cc:290] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.940984 770 raft_consensus.cc:2468] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:21.941586 989 leader_election.cc:304] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:21.942209 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8fcdd6bafa5c4aa6b59668cde523d010" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:21.942409 915 raft_consensus.cc:3060] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.943949 2038 raft_consensus.cc:2804] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:21.944099 2038 raft_consensus.cc:697] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:21.944386 2038 consensus_queue.cc:237] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.944906 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 02e256d4537249a394b80dc36beefef4 (table default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]). Expected version 1 got 0
I20251214 20:52:21.945122 915 raft_consensus.cc:2468] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:21.945247 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
I20251214 20:52:21.946489 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a063083f54ad49d2a842fa080d44ed85" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:21.947001 1977 ts_tablet_manager.cc:1929] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:21.947362 446 catalog_manager.cc:5654] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:21.947952 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:21.949952 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
W20251214 20:52:21.947681 848 leader_election.cc:343] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.950470 1977 log.cc:1199] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/bb698ff30fcc47d09000cc68d2c5225b
I20251214 20:52:21.950971 1977 ts_tablet_manager.cc:1950] T bb698ff30fcc47d09000cc68d2c5225b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:21.951452 1206 tablet_bootstrap.cc:492] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:21.951745 1206 ts_tablet_manager.cc:1403] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20251214 20:52:21.952210 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:21.952683 1202 tablet_bootstrap.cc:492] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:21.952689 1206 raft_consensus.cc:359] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.952910 1206 raft_consensus.cc:385] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.952996 1206 raft_consensus.cc:740] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.953240 1206 consensus_queue.cc:260] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.953429 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:21.953938 1206 ts_tablet_manager.cc:1434] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:21.954097 1982 ts_tablet_manager.cc:1929] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:21.954241 1982 log.cc:1199] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ba50e32f7e724868bb891045a939e362
I20251214 20:52:21.954996 1202 ts_tablet_manager.cc:1403] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.060s user 0.022s sys 0.012s
I20251214 20:52:21.956120 1203 tablet_bootstrap.cc:654] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.956787 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 106) took 1120 ms. Trace:
I20251214 20:52:21.963272 1977 rpcz_store.cc:276] 1214 20:52:20.836091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.836405 (+ 314us) service_pool.cc:225] Handling call
1214 20:52:21.878817 (+1042412us) ts_tablet_manager.cc:1151] Deleting tablet bb698ff30fcc47d09000cc68d2c5225b
1214 20:52:21.878830 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.898328 (+ 19498us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.946854 (+ 48526us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.956742 (+ 9888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019754}
I20251214 20:52:21.964082 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet bb698ff30fcc47d09000cc68d2c5225b (table default.Table_7c95904dd8b044e4ad472681d5a5a274 [id=8c96acfc06ea4f00996fa066d719e418]) successfully deleted
I20251214 20:52:21.964023 1977 tablet_replica.cc:333] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:21.964507 1977 raft_consensus.cc:2243] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:21.964701 1977 raft_consensus.cc:2272] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:21.965334 1977 ts_tablet_manager.cc:1916] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:21.965454 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:21.965709 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8e11723c042a4637839f1b13bed0935a (table default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]). Expected version 1 got 0
I20251214 20:52:21.965811 1982 ts_tablet_manager.cc:1950] T ba50e32f7e724868bb891045a939e362 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:21.966153 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:21.966498 446 catalog_manager.cc:5654] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:21.967733 770 raft_consensus.cc:3060] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:21.969863 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d01db63fd5d2474a9defdce0b98fc29d (table default.Table_13f016135de44cf78187fbc4f9f8c622 [id=fc160acc971e4f7d8290560ee8aab065]). Expected version 1 got 0
I20251214 20:52:21.970388 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:21.972528 1206 tablet_bootstrap.cc:492] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:21.956842 1202 raft_consensus.cc:359] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.972762 1202 raft_consensus.cc:385] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.972873 1202 raft_consensus.cc:740] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.973086 1202 consensus_queue.cc:260] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.973397 2044 raft_consensus.cc:493] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.973668 2044 raft_consensus.cc:515] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:21.974507 2044 leader_election.cc:290] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:21.975334 848 leader_election.cc:343] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 6381cbf05a8c4f84b71f229497fd28ef
I20251214 20:52:21.972167 1203 tablet_bootstrap.cc:492] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:21.975638 1206 tablet_bootstrap.cc:654] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:21.975672 1203 ts_tablet_manager.cc:1403] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.001s
I20251214 20:52:21.976524 1203 raft_consensus.cc:359] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.976713 1203 raft_consensus.cc:385] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:21.976764 1203 raft_consensus.cc:740] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:21.976948 1203 consensus_queue.cc:260] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.973673 1202 ts_tablet_manager.cc:1434] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.019s user 0.000s sys 0.001s
I20251214 20:52:21.979245 1202 tablet_bootstrap.cc:492] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:21.980682 851 leader_election.cc:343] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 6381cbf05a8c4f84b71f229497fd28ef
I20251214 20:52:21.980822 851 leader_election.cc:304] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:21.980839 2079 raft_consensus.cc:493] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:21.980962 2079 raft_consensus.cc:515] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.981591 2079 leader_election.cc:290] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.982649 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 96) took 1818 ms. Trace:
I20251214 20:52:21.982587 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a7fc15882e2474bb6885070be13b1fa" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:21.983498 1982 rpcz_store.cc:276] 1214 20:52:20.164211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.164789 (+ 578us) service_pool.cc:225] Handling call
1214 20:52:21.891268 (+1726479us) ts_tablet_manager.cc:1151] Deleting tablet ba50e32f7e724868bb891045a939e362
1214 20:52:21.891282 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.906487 (+ 15205us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.946913 (+ 40426us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:21.982594 (+ 35681us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1726233}
I20251214 20:52:21.984063 1203 ts_tablet_manager.cc:1434] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251214 20:52:21.984395 1203 tablet_bootstrap.cc:492] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:21.984651 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 98) took 1799 ms. Trace:
I20251214 20:52:21.984994 1982 rpcz_store.cc:276] 1214 20:52:20.184758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.184954 (+ 196us) service_pool.cc:225] Handling call
1214 20:52:21.984306 (+1799352us) ts_tablet_manager.cc:1151] Deleting tablet 8d9f74fb2c6244a6887e3c922262bbab
1214 20:52:21.984513 (+ 207us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.984606 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1798978}
W20251214 20:52:21.985967 989 leader_election.cc:343] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:21.986146 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ba50e32f7e724868bb891045a939e362 (table default.Table_8e3f7e1435ea493fa4729ca0d9e23573 [id=c3e5006d02fc4b69987feb30ba66d069]) successfully deleted
I20251214 20:52:21.986356 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 99) took 1801 ms. Trace:
W20251214 20:52:21.986464 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8d9f74fb2c6244a6887e3c922262bbab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9f74fb2c6244a6887e3c922262bbab already in progress: creating tablet
I20251214 20:52:21.970892 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 1 got 0
I20251214 20:52:21.986871 1982 rpcz_store.cc:276] 1214 20:52:20.184901 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.215783 (+ 30882us) service_pool.cc:225] Handling call
1214 20:52:21.985716 (+1769933us) ts_tablet_manager.cc:1151] Deleting tablet b2e83bca8ed642b6a1b705ef70fde27f
1214 20:52:21.985725 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.986057 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1614278}
I20251214 20:52:21.987142 446 catalog_manager.cc:5654] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
W20251214 20:52:21.987762 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b2e83bca8ed642b6a1b705ef70fde27f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2e83bca8ed642b6a1b705ef70fde27f already in progress: creating tablet
I20251214 20:52:21.987560 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:21.988309 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 0c973e0111804b8b869068bdcc5869bf
I20251214 20:52:21.988366 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 2d164747cb294bd4b126b4420231867d
I20251214 20:52:21.988967 453 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 142) took 1123 ms. Trace:
I20251214 20:52:21.989048 447 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 145) took 1113 ms. Trace:
I20251214 20:52:21.989310 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet e16fe751f10f4616a00d0af811998729
I20251214 20:52:21.990023 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_bb0aa14e61414a5c95c2923827b7cae7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:21.989092 453 rpcz_store.cc:276] 1214 20:52:20.865198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.865272 (+ 74us) service_pool.cc:225] Handling call
1214 20:52:20.865800 (+ 528us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:20.865936 (+ 136us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:20.866030 (+ 94us) catalog_manager.cc:2824] Locking tablets
1214 20:52:21.110686 (+244656us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:21.117909 (+ 7223us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:21.204229 (+ 86320us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:21.988238 (+784009us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:21.988912 (+ 674us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:21.112267 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:21.112428 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:21.112435 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:21.112438 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:21.112585 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:21.112602 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:21.112607 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:21.112745 (+ 138us) tablet.cc:699] Row locks acquired
1214 20:52:21.112748 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:21.112798 (+ 50us) write_op.cc:270] Start()
1214 20:52:21.112839 (+ 41us) write_op.cc:276] Timestamp: P: 1765745541112770 usec, L: 0
1214 20:52:21.112843 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:21.113064 (+ 221us) log.cc:844] Serialized 756 byte log entry
1214 20:52:21.116781 (+ 3717us) op_driver.cc:464] REPLICATION: finished
1214 20:52:21.116981 (+ 200us) write_op.cc:301] APPLY: starting
1214 20:52:21.117001 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:21.117064 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:21.117093 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:21.117228 (+ 135us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:21.117276 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:21.117285 (+ 9us) write_op.cc:312] APPLY: finished
1214 20:52:21.117382 (+ 97us) log.cc:844] Serialized 29 byte log entry
1214 20:52:21.117464 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:21.117491 (+ 27us) write_op.cc:454] Released schema lock
1214 20:52:21.117499 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:21.117591 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:21.117595 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":123,"apply.run_cpu_time_us":555,"apply.run_wall_time_us":588,"num_ops":2,"prepare.queue_time_us":592,"prepare.run_cpu_time_us":942,"prepare.run_wall_time_us":1748,"raft.queue_time_us":409,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":284,"replication_time_us":3905}]]}
W20251214 20:52:21.990619 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb0aa14e61414a5c95c2923827b7cae7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:21.990656 823 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:21.991035 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_0e2b5f246065408b923f196c464eceb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:21.991743 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e2b5f246065408b923f196c464eceb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:21.991864 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 17550eb0ee6b476f8a8c304451f678b2
I20251214 20:52:21.992583 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_2ae9b9f412384e96bfa1e3e6d26ea104" } new_table_name: "default.Table_41d7d906f2714c57832f29b1a2692574" modify_external_catalogs: true
I20251214 20:52:21.993654 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 8eb1006b0ba440dcba2f9de04dca2702
I20251214 20:52:21.994074 451 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 152) took 1008 ms. Trace:
I20251214 20:52:21.994156 451 rpcz_store.cc:276] 1214 20:52:20.985228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.030346 (+ 45118us) service_pool.cc:225] Handling call
1214 20:52:21.031172 (+ 826us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:21.031408 (+ 236us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:21.031503 (+ 95us) catalog_manager.cc:2824] Locking tablets
1214 20:52:21.207870 (+176367us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:21.369239 (+161369us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:21.384883 (+ 15644us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:21.993609 (+608726us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:21.994023 (+ 414us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:21.210088 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:21.210309 (+ 221us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:21.210316 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:21.210318 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:21.210515 (+ 197us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:21.210530 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:21.210535 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:21.210652 (+ 117us) tablet.cc:699] Row locks acquired
1214 20:52:21.210654 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:21.210703 (+ 49us) write_op.cc:270] Start()
1214 20:52:21.210805 (+ 102us) write_op.cc:276] Timestamp: P: 1765745541210655 usec, L: 0
1214 20:52:21.210809 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:21.211000 (+ 191us) log.cc:844] Serialized 725 byte log entry
1214 20:52:21.367014 (+156014us) op_driver.cc:464] REPLICATION: finished
1214 20:52:21.367942 (+ 928us) write_op.cc:301] APPLY: starting
1214 20:52:21.367951 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:21.367982 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:21.368004 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:21.368049 (+ 45us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:21.368075 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:21.368082 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:21.368130 (+ 48us) log.cc:844] Serialized 29 byte log entry
1214 20:52:21.368172 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:21.368184 (+ 12us) write_op.cc:454] Released schema lock
1214 20:52:21.368189 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:21.368246 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:21.368249 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":898,"apply.run_cpu_time_us":298,"apply.run_wall_time_us":298,"num_ops":2,"prepare.queue_time_us":1578,"prepare.run_cpu_time_us":843,"prepare.run_wall_time_us":1634,"raft.queue_time_us":100,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":3,"replication_time_us":156188}]]}
I20251214 20:52:21.994894 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_1e0e46b1f1e1418eadafc528585358f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:21.995260 1964 raft_consensus.cc:493] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:21.995431 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e0e46b1f1e1418eadafc528585358f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:21.995445 1964 raft_consensus.cc:515] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:21.996078 1964 leader_election.cc:290] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:21.999274 449 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_9151bdce51684c39901786aef1c63ec5 [id=f4157d7b9fc848a8b7fa08742b456167] alter complete (version 1)
I20251214 20:52:22.001413 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d164747cb294bd4b126b4420231867d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.001746 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet e16fe751f10f4616a00d0af811998729 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.001842 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 8eb1006b0ba440dcba2f9de04dca2702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.001929 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 17550eb0ee6b476f8a8c304451f678b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.001760 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c973e0111804b8b869068bdcc5869bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.004951 1856 ts_tablet_manager.cc:1929] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.005079 1856 log.cc:1199] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/c9b7149a436a4cab80b39adccf8567f9
I20251214 20:52:22.005416 1856 ts_tablet_manager.cc:1950] T c9b7149a436a4cab80b39adccf8567f9 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:21.989173 447 rpcz_store.cc:276] 1214 20:52:20.875450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.876196 (+ 746us) service_pool.cc:225] Handling call
1214 20:52:20.879090 (+ 2894us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:20.879301 (+ 211us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:20.879391 (+ 90us) catalog_manager.cc:2824] Locking tablets
1214 20:52:21.128729 (+249338us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:21.169817 (+ 41088us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:21.199352 (+ 29535us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:21.988251 (+788899us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:21.988976 (+ 725us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:21.130160 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:21.130502 (+ 342us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:21.130510 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:21.130512 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:21.130750 (+ 238us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:21.130769 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:21.130774 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:21.130971 (+ 197us) tablet.cc:699] Row locks acquired
1214 20:52:21.130996 (+ 25us) write_op.cc:260] PREPARE: finished
1214 20:52:21.131062 (+ 66us) write_op.cc:270] Start()
1214 20:52:21.131151 (+ 89us) write_op.cc:276] Timestamp: P: 1765745541131012 usec, L: 0
1214 20:52:21.131155 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:21.131505 (+ 350us) log.cc:844] Serialized 756 byte log entry
1214 20:52:21.168955 (+ 37450us) op_driver.cc:464] REPLICATION: finished
1214 20:52:21.169135 (+ 180us) write_op.cc:301] APPLY: starting
1214 20:52:21.169158 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:21.169220 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:21.169261 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:21.169375 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:21.169433 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:21.169440 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:21.169572 (+ 132us) log.cc:844] Serialized 29 byte log entry
1214 20:52:21.169645 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:21.169676 (+ 31us) write_op.cc:454] Released schema lock
1214 20:52:21.169686 (+ 10us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:21.169781 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:21.169785 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":638,"apply.run_wall_time_us":637,"num_ops":2,"prepare.queue_time_us":493,"prepare.run_cpu_time_us":1300,"prepare.run_wall_time_us":2721,"raft.queue_time_us":547,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":378,"replication_time_us":37756}]]}
I20251214 20:52:22.006278 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 89) took 2947 ms. Trace:
I20251214 20:52:22.006402 447 master_service.cc:954] ReplaceTablet: received request to replace tablet dfe50a6219134431b0a025317932609b from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:22.006351 1856 rpcz_store.cc:276] 1214 20:52:19.059336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.059387 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:21.882193 (+2822806us) ts_tablet_manager.cc:1151] Deleting tablet c9b7149a436a4cab80b39adccf8567f9
1214 20:52:21.882203 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:21.927212 (+ 45009us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.004795 (+ 77583us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.006256 (+ 1461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2762965}
I20251214 20:52:22.007599 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet c9b7149a436a4cab80b39adccf8567f9 (table default.Table_67ec8d0cd3f948218b4637de6d68a57b [id=bdc2d86a977d4146a1a86717c5916807]) successfully deleted
I20251214 20:52:22.007843 1856 tablet_replica.cc:333] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:22.008064 1856 raft_consensus.cc:2243] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.008224 1856 raft_consensus.cc:2272] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.008930 1856 ts_tablet_manager.cc:1916] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.009086 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d164747cb294bd4b126b4420231867d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.009367 743 tablet_service.cc:1552] Processing DeleteTablet for tablet e16fe751f10f4616a00d0af811998729 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.009438 757 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c973e0111804b8b869068bdcc5869bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.009595 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 17550eb0ee6b476f8a8c304451f678b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.009778 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 8eb1006b0ba440dcba2f9de04dca2702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.010394 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fcfcfe81fe24d369eea696efc61d21c" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:22.010607 1056 raft_consensus.cc:2468] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:22.013113 1203 tablet_bootstrap.cc:654] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:22.014597 713 consensus_peers.cc:597] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c0f49e5d78240a296e47eccb3218e4f. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:22.014798 713 leader_election.cc:304] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:22.015161 1964 raft_consensus.cc:2804] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:22.015277 1964 raft_consensus.cc:493] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:22.015378 1964 raft_consensus.cc:3060] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.018168 1964 raft_consensus.cc:515] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.018903 1964 leader_election.cc:290] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.019423 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fcfcfe81fe24d369eea696efc61d21c" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:22.019690 1856 ts_tablet_manager.cc:1929] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:22.019817 1856 log.cc:1199] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/9d89ee602c524c96a21c0a235af07481
I20251214 20:52:22.019704 1056 raft_consensus.cc:3060] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.020232 1856 ts_tablet_manager.cc:1950] T 9d89ee602c524c96a21c0a235af07481 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:22.021200 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 97) took 2747 ms. Trace:
I20251214 20:52:22.021284 1856 rpcz_store.cc:276] 1214 20:52:19.273862 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.274105 (+ 243us) service_pool.cc:225] Handling call
1214 20:52:22.007777 (+2733672us) ts_tablet_manager.cc:1151] Deleting tablet 9d89ee602c524c96a21c0a235af07481
1214 20:52:22.007788 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.012127 (+ 4339us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.014974 (+ 2847us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.021179 (+ 6205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2732094}
I20251214 20:52:22.021785 1856 tablet_replica.cc:333] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:22.021737 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 9d89ee602c524c96a21c0a235af07481 (table default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]) successfully deleted
I20251214 20:52:22.022001 1856 raft_consensus.cc:2243] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:22.022289 1856 pending_rounds.cc:70] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:22.022323 1856 pending_rounds.cc:77] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493728858238976 op_type: NO_OP noop_request { }
I20251214 20:52:22.022404 1856 raft_consensus.cc:2889] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:22.022469 1856 raft_consensus.cc:2272] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.022997 1856 ts_tablet_manager.cc:1916] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.023182 1056 raft_consensus.cc:2468] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:22.023766 713 leader_election.cc:304] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:22.024117 1964 raft_consensus.cc:2804] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:22.024300 1964 raft_consensus.cc:697] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:22.024575 1964 consensus_queue.cc:237] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.027410 1203 tablet_bootstrap.cc:492] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:22.027570 1203 ts_tablet_manager.cc:1403] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.003s
I20251214 20:52:22.028380 1203 raft_consensus.cc:359] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.028554 1203 raft_consensus.cc:385] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.028614 1203 raft_consensus.cc:740] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.028854 1203 consensus_queue.cc:260] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.031093 1203 ts_tablet_manager.cc:1434] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251214 20:52:22.031430 1203 tablet_bootstrap.cc:492] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:22.032796 1856 ts_tablet_manager.cc:1929] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.032915 1856 log.cc:1199] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/fe4086fc7a4f4db0b7541f06c9833aab
I20251214 20:52:22.033245 1203 tablet_bootstrap.cc:654] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.033265 1856 ts_tablet_manager.cc:1950] T fe4086fc7a4f4db0b7541f06c9833aab P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:22.034126 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 110) took 2153 ms. Trace:
I20251214 20:52:22.034215 1856 rpcz_store.cc:276] 1214 20:52:19.881080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.881215 (+ 135us) service_pool.cc:225] Handling call
1214 20:52:22.021730 (+2140515us) ts_tablet_manager.cc:1151] Deleting tablet fe4086fc7a4f4db0b7541f06c9833aab
1214 20:52:22.021741 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.027620 (+ 5879us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.032691 (+ 5071us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.034105 (+ 1414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2140220}
I20251214 20:52:22.034770 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 111) took 2112 ms. Trace:
I20251214 20:52:22.034792 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet fe4086fc7a4f4db0b7541f06c9833aab (table default.Table_8076bc8b31ba411faf089f314e6ed2ef [id=25016f04bb864f92a1cdfb6ceaa9fba9]) successfully deleted
I20251214 20:52:22.034858 1856 rpcz_store.cc:276] 1214 20:52:19.922556 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.922723 (+ 167us) service_pool.cc:225] Handling call
1214 20:52:22.034636 (+2111913us) ts_tablet_manager.cc:1151] Deleting tablet 98b7cac3930a4a9eb16ac413c8f1e092
1214 20:52:22.034646 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.034756 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2105865}
I20251214 20:52:22.035327 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 112) took 2100 ms. Trace:
W20251214 20:52:22.035311 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 98b7cac3930a4a9eb16ac413c8f1e092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98b7cac3930a4a9eb16ac413c8f1e092 already in progress: creating tablet
I20251214 20:52:22.035405 1856 rpcz_store.cc:276] 1214 20:52:19.935338 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.935407 (+ 69us) service_pool.cc:225] Handling call
1214 20:52:22.035231 (+2099824us) ts_tablet_manager.cc:1151] Deleting tablet 7dd657f3baff40bb8cb7f4648367f7a1
1214 20:52:22.035240 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.035311 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2088329}
I20251214 20:52:22.035781 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 116) took 2004 ms. Trace:
W20251214 20:52:22.035789 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 7dd657f3baff40bb8cb7f4648367f7a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dd657f3baff40bb8cb7f4648367f7a1 already in progress: creating tablet
I20251214 20:52:22.035861 1856 rpcz_store.cc:276] 1214 20:52:20.031700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.033530 (+ 1830us) service_pool.cc:225] Handling call
1214 20:52:22.035696 (+2002166us) ts_tablet_manager.cc:1151] Deleting tablet 292a28b12df246a8bc855d81c530746e
1214 20:52:22.035703 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.035768 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2001958}
I20251214 20:52:22.036185 1856 tablet_replica.cc:333] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:22.036206 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 292a28b12df246a8bc855d81c530746e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292a28b12df246a8bc855d81c530746e already in progress: creating tablet
I20251214 20:52:22.036388 1856 raft_consensus.cc:2243] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:22.036722 1856 raft_consensus.cc:2272] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.037298 1856 ts_tablet_manager.cc:1916] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.037669 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a7fc15882e2474bb6885070be13b1fa" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:22.040849 2117 tablet.cc:1722] T 07a8662a634a4befbd2d6251000d2059 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:22.044961 909 raft_consensus.cc:2468] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:22.046010 991 leader_election.cc:304] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:22.046653 2038 raft_consensus.cc:2804] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:22.046729 2038 raft_consensus.cc:493] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:22.046782 2038 raft_consensus.cc:3060] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.048830 1762 raft_consensus.cc:2749] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:22.049368 2038 raft_consensus.cc:515] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.050846 2038 leader_election.cc:290] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.051860 1856 ts_tablet_manager.cc:1929] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251214 20:52:22.051977 1856 log.cc:1199] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/2b48bb260e854beb9ec61ce9c79b46c2
I20251214 20:52:22.052317 1856 ts_tablet_manager.cc:1950] T 2b48bb260e854beb9ec61ce9c79b46c2 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:22.053040 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c973e0111804b8b869068bdcc5869bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.053181 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 117) took 2009 ms. Trace:
I20251214 20:52:22.053249 1856 rpcz_store.cc:276] 1214 20:52:20.043681 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.043764 (+ 83us) service_pool.cc:225] Handling call
1214 20:52:22.036141 (+1992377us) ts_tablet_manager.cc:1151] Deleting tablet 2b48bb260e854beb9ec61ce9c79b46c2
1214 20:52:22.036148 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.045197 (+ 9049us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.051728 (+ 6531us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.053161 (+ 1433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1991777}
I20251214 20:52:22.053764 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 2b48bb260e854beb9ec61ce9c79b46c2 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]) successfully deleted
I20251214 20:52:22.053831 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 118) took 1997 ms. Trace:
I20251214 20:52:22.053907 1856 rpcz_store.cc:276] 1214 20:52:20.056611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.056660 (+ 49us) service_pool.cc:225] Handling call
1214 20:52:22.053696 (+1997036us) ts_tablet_manager.cc:1151] Deleting tablet 242a77ae57174de48f4eaf43384ee0ba
1214 20:52:22.053705 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.053815 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1996716}
I20251214 20:52:22.054268 1203 tablet_bootstrap.cc:492] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
W20251214 20:52:22.054270 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 242a77ae57174de48f4eaf43384ee0ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 242a77ae57174de48f4eaf43384ee0ba already in progress: creating tablet
I20251214 20:52:22.054388 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 119) took 1913 ms. Trace:
I20251214 20:52:22.054454 1856 rpcz_store.cc:276] 1214 20:52:20.140663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.140738 (+ 75us) service_pool.cc:225] Handling call
1214 20:52:22.054219 (+1913481us) ts_tablet_manager.cc:1151] Deleting tablet 65e79db74fa046698507c83416ee1669
1214 20:52:22.054226 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.054374 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1913216}
I20251214 20:52:22.054806 1856 tablet_replica.cc:333] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:22.054908 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 65e79db74fa046698507c83416ee1669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e79db74fa046698507c83416ee1669 already in progress: creating tablet
I20251214 20:52:22.054996 1856 raft_consensus.cc:2243] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:22.055341 1856 raft_consensus.cc:2272] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.055539 894 tablet_service.cc:1552] Processing DeleteTablet for tablet e16fe751f10f4616a00d0af811998729 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.042374 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 121) took 1673 ms. Trace:
I20251214 20:52:22.055923 1856 ts_tablet_manager.cc:1916] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.055971 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 17550eb0ee6b476f8a8c304451f678b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.054407 1203 ts_tablet_manager.cc:1403] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251214 20:52:22.056176 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d164747cb294bd4b126b4420231867d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.056397 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a7fc15882e2474bb6885070be13b1fa" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:22.056852 1203 raft_consensus.cc:359] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.057011 1203 raft_consensus.cc:385] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.057054 1203 raft_consensus.cc:740] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.057152 909 raft_consensus.cc:3060] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.057243 1203 consensus_queue.cc:260] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.057453 1982 rpcz_store.cc:276] 1214 20:52:20.368869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.371561 (+ 2692us) service_pool.cc:225] Handling call
1214 20:52:21.987788 (+1616227us) ts_tablet_manager.cc:1151] Deleting tablet 06381efe75284e649fd6d1a9903cfa9b
1214 20:52:21.987797 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.042299 (+ 54502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1615990}
W20251214 20:52:22.058256 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 06381efe75284e649fd6d1a9903cfa9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06381efe75284e649fd6d1a9903cfa9b already in progress: creating tablet
I20251214 20:52:22.058429 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 122) took 1689 ms. Trace:
I20251214 20:52:22.058539 1982 rpcz_store.cc:276] 1214 20:52:20.369104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.371817 (+ 2713us) service_pool.cc:225] Handling call
1214 20:52:22.058240 (+1686423us) ts_tablet_manager.cc:1151] Deleting tablet b1f4ea72c6274569a98cab6afdc55d55
1214 20:52:22.058253 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.058362 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1686257}
I20251214 20:52:22.059476 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 123) took 1690 ms. Trace:
W20251214 20:52:22.059536 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b1f4ea72c6274569a98cab6afdc55d55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1f4ea72c6274569a98cab6afdc55d55 already in progress: creating tablet
I20251214 20:52:22.059820 1202 tablet_bootstrap.cc:654] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.059950 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9e89276f2f2b4a7981f91242bdb24a87" } new_table_name: "default.Table_89c0478b97144f35b1a0adeb64be2d9f" modify_external_catalogs: true
I20251214 20:52:22.061236 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 8eb1006b0ba440dcba2f9de04dca2702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.061448 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b0d8d299bd0b4950948be859b03a3731" } new_table_name: "default.Table_55f303d33ec84a269232fc1ba8f2d182" modify_external_catalogs: true
I20251214 20:52:22.062214 1856 ts_tablet_manager.cc:1929] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:22.062321 1856 log.cc:1199] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/25974e5db555447baf5799775d15cd14
I20251214 20:52:22.062677 1856 ts_tablet_manager.cc:1950] T 25974e5db555447baf5799775d15cd14 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:22.063247 961 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
W20251214 20:52:22.065574 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251214 20:52:22.066686 442 consensus_peers.cc:597] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a -> Peer e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819): Couldn't send request to peer e6713a20a82b434f8a8bc122113e26ad. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251214 20:52:22.067236 1982 rpcz_store.cc:276] 1214 20:52:20.369275 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.371989 (+ 2714us) service_pool.cc:225] Handling call
1214 20:52:22.059273 (+1687284us) ts_tablet_manager.cc:1151] Deleting tablet 4d6e7306254044449f1127fb963ca5f0
1214 20:52:22.059282 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.059412 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1687138}
W20251214 20:52:22.068035 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 4d6e7306254044449f1127fb963ca5f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d6e7306254044449f1127fb963ca5f0 already in progress: creating tablet
I20251214 20:52:22.068101 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 124) took 1698 ms. Trace:
I20251214 20:52:22.068181 1982 rpcz_store.cc:276] 1214 20:52:20.369356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.372138 (+ 2782us) service_pool.cc:225] Handling call
1214 20:52:22.067768 (+1695630us) ts_tablet_manager.cc:1151] Deleting tablet 69f6108792fa4c2a9744e3248c76251f
1214 20:52:22.067779 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.068044 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1695510}
W20251214 20:52:22.051988 712 leader_election.cc:343] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 7fcfcfe81fe24d369eea696efc61d21c
W20251214 20:52:22.068886 712 leader_election.cc:343] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 7fcfcfe81fe24d369eea696efc61d21c
I20251214 20:52:22.043179 1206 tablet_bootstrap.cc:492] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:22.069687 1206 ts_tablet_manager.cc:1403] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.001s
I20251214 20:52:22.070043 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 123) took 1670 ms. Trace:
I20251214 20:52:22.070147 1856 rpcz_store.cc:276] 1214 20:52:20.399257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.399493 (+ 236us) service_pool.cc:225] Handling call
1214 20:52:22.054760 (+1655267us) ts_tablet_manager.cc:1151] Deleting tablet 25974e5db555447baf5799775d15cd14
1214 20:52:22.054769 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.059080 (+ 4311us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.062105 (+ 3025us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.070019 (+ 7914us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6201,"tablet-delete.queue_time_us":1655026}
I20251214 20:52:22.070375 909 raft_consensus.cc:2468] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:22.070560 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]) successfully deleted
W20251214 20:52:22.070606 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 69f6108792fa4c2a9744e3248c76251f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f6108792fa4c2a9744e3248c76251f already in progress: creating tablet
I20251214 20:52:22.070114 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 125) took 1700 ms. Trace:
I20251214 20:52:22.071095 991 leader_election.cc:304] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.071255 1982 rpcz_store.cc:276] 1214 20:52:20.369443 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.372267 (+ 2824us) service_pool.cc:225] Handling call
1214 20:52:22.069966 (+1697699us) ts_tablet_manager.cc:1151] Deleting tablet 9d89ee602c524c96a21c0a235af07481
1214 20:52:22.069976 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.070067 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1697555}
I20251214 20:52:22.071411 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 131) took 1397 ms. Trace:
I20251214 20:52:22.071460 2038 raft_consensus.cc:2804] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:22.071492 1856 rpcz_store.cc:276] 1214 20:52:20.673854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.673920 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:22.071250 (+1397330us) ts_tablet_manager.cc:1151] Deleting tablet 2accf8a160584c82bb695a52e3f2b05c
1214 20:52:22.071263 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.071395 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1397077}
I20251214 20:52:22.071580 2038 raft_consensus.cc:697] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
W20251214 20:52:22.071854 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2accf8a160584c82bb695a52e3f2b05c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2accf8a160584c82bb695a52e3f2b05c already in progress: creating tablet
I20251214 20:52:22.071866 2038 consensus_queue.cc:237] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.072144 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 101) took 1855 ms. Trace:
I20251214 20:52:22.072371 1856 tablet_replica.cc:333] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:22.072472 1982 rpcz_store.cc:276] 1214 20:52:20.216268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.216314 (+ 46us) service_pool.cc:225] Handling call
1214 20:52:22.071612 (+1855298us) ts_tablet_manager.cc:1151] Deleting tablet ee7d709316b24b8eb223a838710c2979
1214 20:52:22.071620 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.072097 (+ 477us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1699132}
W20251214 20:52:22.073169 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 9d89ee602c524c96a21c0a235af07481 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d89ee602c524c96a21c0a235af07481 already in progress: creating tablet
I20251214 20:52:22.075433 1203 ts_tablet_manager.cc:1434] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251214 20:52:22.075762 1203 tablet_bootstrap.cc:492] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:22.072721 1206 raft_consensus.cc:359] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:22.077996 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ee7d709316b24b8eb223a838710c2979 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7d709316b24b8eb223a838710c2979 already in progress: creating tablet
I20251214 20:52:22.078146 1206 raft_consensus.cc:385] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.078295 1206 raft_consensus.cc:740] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.078606 1206 consensus_queue.cc:260] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.079375 1203 tablet_bootstrap.cc:654] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.080736 1206 ts_tablet_manager.cc:1434] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251214 20:52:22.085559 1203 tablet_bootstrap.cc:492] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:22.085716 1203 ts_tablet_manager.cc:1403] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20251214 20:52:22.086537 1203 raft_consensus.cc:359] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.086728 1203 raft_consensus.cc:385] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.086796 1203 raft_consensus.cc:740] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.087039 1203 consensus_queue.cc:260] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.089283 1203 ts_tablet_manager.cc:1434] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:22.089600 1203 tablet_bootstrap.cc:492] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:22.091697 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 102) took 1871 ms. Trace:
I20251214 20:52:22.090605 1977 ts_tablet_manager.cc:1929] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.095188 1977 log.cc:1199] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/60ae3db968f6412aa0c3952fafe65f88
I20251214 20:52:22.095659 1977 ts_tablet_manager.cc:1950] T 60ae3db968f6412aa0c3952fafe65f88 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:22.096928 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 108) took 1260 ms. Trace:
I20251214 20:52:22.101616 1982 rpcz_store.cc:276] 1214 20:52:20.220258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.220311 (+ 53us) service_pool.cc:225] Handling call
1214 20:52:22.078000 (+1857689us) ts_tablet_manager.cc:1151] Deleting tablet 604b7c6fe3b24daca70711a1990af556
1214 20:52:22.084873 (+ 6873us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.091625 (+ 6752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1705460}
I20251214 20:52:22.095934 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a7fc15882e2474bb6885070be13b1fa" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:22.102378 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 604b7c6fe3b24daca70711a1990af556 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 604b7c6fe3b24daca70711a1990af556 already in progress: creating tablet
W20251214 20:52:22.102829 989 leader_election.cc:343] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.103576 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 103) took 1882 ms. Trace:
I20251214 20:52:22.103785 1982 rpcz_store.cc:276] 1214 20:52:20.220681 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.220734 (+ 53us) service_pool.cc:225] Handling call
1214 20:52:22.103401 (+1882667us) ts_tablet_manager.cc:1151] Deleting tablet f84e9dfc23d34eb0a379e8ac5a442e59
1214 20:52:22.103415 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.103520 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1730820}
W20251214 20:52:22.104281 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f84e9dfc23d34eb0a379e8ac5a442e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f84e9dfc23d34eb0a379e8ac5a442e59 already in progress: creating tablet
I20251214 20:52:22.104321 1982 tablet_replica.cc:333] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:22.104640 1982 raft_consensus.cc:2243] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:22.105819 1982 raft_consensus.cc:2272] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.106887 1982 ts_tablet_manager.cc:1916] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.109103 1203 tablet_bootstrap.cc:654] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.109841 1856 raft_consensus.cc:2243] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.110136 1856 raft_consensus.cc:2272] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.111374 1977 rpcz_store.cc:276] 1214 20:52:20.836306 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.836351 (+ 45us) service_pool.cc:225] Handling call
1214 20:52:21.963940 (+1127589us) ts_tablet_manager.cc:1151] Deleting tablet 60ae3db968f6412aa0c3952fafe65f88
1214 20:52:21.963963 (+ 23us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.075289 (+111326us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.090444 (+ 15155us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.096872 (+ 6428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106299}
I20251214 20:52:22.112402 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 60ae3db968f6412aa0c3952fafe65f88 (table default.Table_01dcfe83d7854b4e9b8d6f62a557a0fe [id=d180e3a95c0840589c0aeec13ef1a6ca]) successfully deleted
I20251214 20:52:22.114418 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 112) took 1277 ms. Trace:
I20251214 20:52:22.114696 1977 rpcz_store.cc:276] 1214 20:52:20.836887 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.836930 (+ 43us) service_pool.cc:225] Handling call
1214 20:52:22.114247 (+1277317us) ts_tablet_manager.cc:1151] Deleting tablet 40a9a6192c744282ab5672cbe15fd299
1214 20:52:22.114255 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.114351 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1250930}
I20251214 20:52:22.115728 1856 ts_tablet_manager.cc:1916] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:22.116218 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 40a9a6192c744282ab5672cbe15fd299 because the tablet was not found. No further retry: Not found: Tablet not found: 40a9a6192c744282ab5672cbe15fd299
I20251214 20:52:22.116609 1977 tablet_replica.cc:333] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:22.116865 1977 raft_consensus.cc:2243] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.117666 1203 tablet_bootstrap.cc:492] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:22.117810 1203 ts_tablet_manager.cc:1403] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20251214 20:52:22.118148 1977 raft_consensus.cc:2272] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.118707 1203 raft_consensus.cc:359] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.118888 1203 raft_consensus.cc:385] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.118942 1203 raft_consensus.cc:740] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.119194 1203 consensus_queue.cc:260] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:22.119729 454 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6bdfbe4d15345de8be5de00bfe4ca07 consensus state has no leader
I20251214 20:52:22.119935 454 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 160) took 1041 ms. Trace:
I20251214 20:52:22.120023 454 rpcz_store.cc:276] 1214 20:52:21.078132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.992394 (+914262us) service_pool.cc:225] Handling call
1214 20:52:21.992770 (+ 376us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:21.992987 (+ 217us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:21.993170 (+ 183us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:21.993241 (+ 71us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:22.119448 (+126207us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:22.119874 (+ 426us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.105457 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.112863 (+ 7406us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.112870 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:22.112873 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.113000 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.113016 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.113021 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.113092 (+ 71us) tablet.cc:699] Row locks acquired
1214 20:52:22.113095 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.113133 (+ 38us) write_op.cc:270] Start()
1214 20:52:22.113167 (+ 34us) write_op.cc:276] Timestamp: P: 1765745542113112 usec, L: 0
1214 20:52:22.113171 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.113368 (+ 197us) log.cc:844] Serialized 405 byte log entry
1214 20:52:22.118349 (+ 4981us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.118583 (+ 234us) write_op.cc:301] APPLY: starting
1214 20:52:22.118597 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.118639 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.118678 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.118729 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.118764 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.118771 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.118841 (+ 70us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.118898 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.118915 (+ 17us) write_op.cc:454] Released schema lock
1214 20:52:22.118921 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.119002 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.119006 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":188,"apply.run_cpu_time_us":411,"apply.run_wall_time_us":411,"num_ops":1,"prepare.queue_time_us":111887,"prepare.run_cpu_time_us":851,"prepare.run_wall_time_us":9938,"raft.queue_time_us":359,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":287,"replication_time_us":5150}]]}
I20251214 20:52:22.115092 1202 tablet_bootstrap.cc:492] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:22.121029 1202 ts_tablet_manager.cc:1403] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.142s user 0.012s sys 0.009s
I20251214 20:52:22.121321 1203 ts_tablet_manager.cc:1434] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:22.121634 1203 tablet_bootstrap.cc:492] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:22.122329 1202 raft_consensus.cc:359] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.122531 1202 raft_consensus.cc:385] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.122594 1202 raft_consensus.cc:740] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.122829 1202 consensus_queue.cc:260] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.123368 1203 tablet_bootstrap.cc:654] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.123580 1977 ts_tablet_manager.cc:1916] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.126029 1206 tablet_bootstrap.cc:492] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:22.127957 448 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 0d7c7b3f62764ff497f4d70b53eed7de (table default.Table_e26cdfc4ab8f4f19b52de34448bd70e5 [id=756e7314dcb5446f8e62baab11878de2], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0d7c7b3f62764ff497f4d70b53eed7de consensus state has no leader
I20251214 20:52:22.128288 1206 tablet_bootstrap.cc:654] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:22.129479 448 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f consensus state has no leader
I20251214 20:52:22.130232 1762 raft_consensus.cc:493] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.130414 1762 raft_consensus.cc:515] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.131009 1762 leader_election.cc:290] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.132445 1202 ts_tablet_manager.cc:1434] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251214 20:52:22.132759 1202 tablet_bootstrap.cc:492] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:22.133590 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet b1df0dad2fb040a5a12e0401e00c87db with delete_type TABLET_DATA_DELETED (replaced by tablet 8fcdd6bafa5c4aa6b59668cde523d010) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.133772 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 012b81223d0f4ae9b742014e329f11a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.133857 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.133960 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 569a02142c71427d91868b02695ae12b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.133999 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet a10663bd306d40e7a6375d06d69be22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.134111 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b48bb260e854beb9ec61ce9c79b46c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 566c6e4761d5486d964e276d151e32b0) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.134132 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b7149a436a4cab80b39adccf8567f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.134292 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.134464 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet fe4086fc7a4f4db0b7541f06c9833aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.134820 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.134987 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet a0279992d1f2419ab1caae116a0391b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.135151 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.135303 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.135424 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.135540 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 292a28b12df246a8bc855d81c530746e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.134276 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 24ff58f321124d12b5330dca185d426e with delete_type TABLET_DATA_DELETED (replaced by tablet 42bdcecba5534294aca1eb419f18e9d2) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.136607 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d89ee602c524c96a21c0a235af07481" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:22.137333 848 leader_election.cc:343] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:22.139853 851 leader_election.cc:343] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 9d89ee602c524c96a21c0a235af07481
I20251214 20:52:22.140028 851 leader_election.cc:304] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:22.140468 1202 tablet_bootstrap.cc:654] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.144603 1203 tablet_bootstrap.cc:492] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:22.144773 1203 ts_tablet_manager.cc:1403] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.003s
I20251214 20:52:22.140807 1762 raft_consensus.cc:2749] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:22.145546 1982 ts_tablet_manager.cc:1929] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.145597 1203 raft_consensus.cc:359] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.145784 1203 raft_consensus.cc:385] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.145834 1203 raft_consensus.cc:740] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.146067 1203 consensus_queue.cc:260] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.146749 1203 ts_tablet_manager.cc:1434] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:22.147047 1203 tablet_bootstrap.cc:492] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:22.147401 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet b315744c8cd043c5a8bdceec73a7e28b with delete_type TABLET_DATA_DELETED (replaced by tablet f945814952b14131ba4e0c67a403a680) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.148842 1762 raft_consensus.cc:493] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.149044 1762 raft_consensus.cc:515] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.149161 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet c16895ca08d94b56b5c4bc9352e330ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.149503 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet a34eb8fa3e34427495fa5985b7ba81bf with delete_type TABLET_DATA_DELETED (replaced by tablet 6fef1d6add764fd28a8f4995b337ea71) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.149675 1762 leader_election.cc:290] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.149828 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 25974e5db555447baf5799775d15cd14 with delete_type TABLET_DATA_DELETED (replaced by tablet d171301941fa43908e05984c3b016007) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.151564 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35a529702f924a5c9bf53c3c668c6094" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:22.151775 1056 raft_consensus.cc:2468] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:22.152346 851 leader_election.cc:304] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.152716 1762 raft_consensus.cc:2804] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:22.152822 1762 raft_consensus.cc:493] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:22.152917 1762 raft_consensus.cc:3060] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.153736 1977 ts_tablet_manager.cc:1929] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.153858 1977 log.cc:1199] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/c9b7149a436a4cab80b39adccf8567f9
I20251214 20:52:22.154271 1977 ts_tablet_manager.cc:1950] T c9b7149a436a4cab80b39adccf8567f9 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:22.155079 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 111) took 1318 ms. Trace:
I20251214 20:52:22.155215 1977 rpcz_store.cc:276] 1214 20:52:20.836761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.836927 (+ 166us) service_pool.cc:225] Handling call
1214 20:52:22.116528 (+1279601us) ts_tablet_manager.cc:1151] Deleting tablet c9b7149a436a4cab80b39adccf8567f9
1214 20:52:22.116538 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.140769 (+ 24231us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.153578 (+ 12809us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.155033 (+ 1455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251417}
I20251214 20:52:22.155583 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 113) took 1318 ms. Trace:
I20251214 20:52:22.155671 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3962c33a294fa98f8bd485268aee71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.155756 1977 rpcz_store.cc:276] 1214 20:52:20.837034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.837101 (+ 67us) service_pool.cc:225] Handling call
1214 20:52:22.155477 (+1318376us) ts_tablet_manager.cc:1151] Deleting tablet c16895ca08d94b56b5c4bc9352e330ef
1214 20:52:22.155486 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.155548 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1257045}
I20251214 20:52:22.156127 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 115) took 1318 ms. Trace:
I20251214 20:52:22.156140 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet ee7d709316b24b8eb223a838710c2979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.156234 1977 rpcz_store.cc:276] 1214 20:52:20.837207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.837377 (+ 170us) service_pool.cc:225] Handling call
1214 20:52:22.155986 (+1318609us) ts_tablet_manager.cc:1151] Deleting tablet 6bd6856b800247de896f16eee66f2827
1214 20:52:22.155993 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.156093 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255985}
I20251214 20:52:22.156594 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dd657f3baff40bb8cb7f4648367f7a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.157969 1762 raft_consensus.cc:515] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.159631 1762 leader_election.cc:290] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.160298 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35a529702f924a5c9bf53c3c668c6094" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:22.160516 1056 raft_consensus.cc:3060] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.161015 2038 raft_consensus.cc:493] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.161139 2038 raft_consensus.cc:515] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.161496 1856 ts_tablet_manager.cc:1929] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:22.161597 1856 log.cc:1199] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/762068d560ff47918dd9fd8791083433
I20251214 20:52:22.161756 2038 leader_election.cc:290] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.161914 1856 ts_tablet_manager.cc:1950] T 762068d560ff47918dd9fd8791083433 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:22.163183 1203 tablet_bootstrap.cc:654] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.163676 1056 raft_consensus.cc:2468] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:22.164126 851 leader_election.cc:304] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.164474 1762 raft_consensus.cc:2804] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:22.164471 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet c9b7149a436a4cab80b39adccf8567f9 (table default.Table_67ec8d0cd3f948218b4637de6d68a57b [id=bdc2d86a977d4146a1a86717c5916807]) successfully deleted
I20251214 20:52:22.164824 1762 raft_consensus.cc:697] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:22.165812 1762 consensus_queue.cc:237] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:22.165891 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c16895ca08d94b56b5c4bc9352e330ef because the tablet was not found. No further retry: Not found: Tablet not found: c16895ca08d94b56b5c4bc9352e330ef
W20251214 20:52:22.166177 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 6bd6856b800247de896f16eee66f2827 because the tablet was not found. No further retry: Not found: Tablet not found: 6bd6856b800247de896f16eee66f2827
I20251214 20:52:22.167492 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d89ee602c524c96a21c0a235af07481 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e11723c042a4637839f1b13bed0935a) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.167910 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 35a529702f924a5c9bf53c3c668c6094 with delete_type TABLET_DATA_DELETED (replaced by tablet 719d157ac8f74eb7911df05ab5e4a3d5) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.168092 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7d709316b24b8eb223a838710c2979" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:22.168314 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 1dca369dc5f54a4ea1783e334c5670d6) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.168319 909 raft_consensus.cc:2468] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:22.168751 991 leader_election.cc:304] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.168860 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 2accf8a160584c82bb695a52e3f2b05c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.169046 2038 raft_consensus.cc:2804] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:22.169124 2038 raft_consensus.cc:493] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:22.169193 2038 raft_consensus.cc:3060] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.169478 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 116) took 1332 ms. Trace:
I20251214 20:52:22.170789 1202 tablet_bootstrap.cc:492] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:22.171660 1202 ts_tablet_manager.cc:1403] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.002s
I20251214 20:52:22.171725 2038 raft_consensus.cc:515] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.173069 1202 raft_consensus.cc:359] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:22.173368 2038 leader_election.cc:290] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.173408 1202 raft_consensus.cc:385] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.173611 1202 raft_consensus.cc:740] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.174074 1202 consensus_queue.cc:260] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:22.174769 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7d709316b24b8eb223a838710c2979" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:22.174996 909 raft_consensus.cc:3060] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.170722 1977 rpcz_store.cc:276] 1214 20:52:20.837366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.837469 (+ 103us) service_pool.cc:225] Handling call
1214 20:52:22.156443 (+1318974us) ts_tablet_manager.cc:1151] Deleting tablet 324379e3217442c3addcfe58060b9ad7
1214 20:52:22.167943 (+ 11500us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.169349 (+ 1406us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254972}
I20251214 20:52:22.175602 1203 tablet_bootstrap.cc:492] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:22.175892 1203 ts_tablet_manager.cc:1403] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
W20251214 20:52:22.176141 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 324379e3217442c3addcfe58060b9ad7 because the tablet was not found. No further retry: Not found: Tablet not found: 324379e3217442c3addcfe58060b9ad7
I20251214 20:52:22.176765 1203 raft_consensus.cc:359] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.177183 1203 raft_consensus.cc:385] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.177235 1203 raft_consensus.cc:740] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.177486 1203 consensus_queue.cc:260] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.178172 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 65e79db74fa046698507c83416ee1669 with delete_type TABLET_DATA_DELETED (replaced by tablet 4dcd511a5dca4a2cbda81fbc255ce354) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.179176 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f with delete_type TABLET_DATA_DELETED (replaced by tablet f11834cabdad445b8e53cd237d146282) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.179839 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 242a77ae57174de48f4eaf43384ee0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.180661 1203 ts_tablet_manager.cc:1434] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:22.180960 1203 tablet_bootstrap.cc:492] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:22.182408 448 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 727e7b61ae3942098712720d344c5310 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 727e7b61ae3942098712720d344c5310 consensus state has no leader
I20251214 20:52:22.182709 1203 tablet_bootstrap.cc:654] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.183050 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 40a9a6192c744282ab5672cbe15fd299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.149165 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3962c33a294fa98f8bd485268aee71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.150648 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35a529702f924a5c9bf53c3c668c6094" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:22.159394 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35a529702f924a5c9bf53c3c668c6094" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:22.162475 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7d709316b24b8eb223a838710c2979" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:22.163265 1964 raft_consensus.cc:493] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.168061 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bbb28259c1442ae94f4eff76c25c539 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.168200 757 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b7149a436a4cab80b39adccf8567f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.173354 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7d709316b24b8eb223a838710c2979" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:22.174417 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.176923 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.177840 740 tablet_service.cc:1552] Processing DeleteTablet for tablet c2ae9dc8ead8403cb7215af8778872f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:22.187410 989 leader_election.cc:343] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.179561 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af99e0bd8774a678575847a9ec2e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.181525 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff with delete_type TABLET_DATA_DELETED (replaced by tablet 6a000951f61948ac85fdd431f3d162c2) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.182053 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 52daf1204bf142e7b82852bdec6582fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.182638 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a7fc15882e2474bb6885070be13b1fa with delete_type TABLET_DATA_DELETED (replaced by tablet 65e79db74fa046698507c83416ee1669) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.184284 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 9523d72fd91445539d67e1eac2a80afd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.186022 753 tablet_service.cc:1552] Processing DeleteTablet for tablet fe4086fc7a4f4db0b7541f06c9833aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.186350 1982 log.cc:1199] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/b1df0dad2fb040a5a12e0401e00c87db
W20251214 20:52:22.189180 448 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:46855 (request call id 162) took 1988 ms (client timeout 1000 ms). Trace:
I20251214 20:52:22.193271 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 130) took 1529 ms. Trace:
I20251214 20:52:22.193393 1856 rpcz_store.cc:276] 1214 20:52:20.663832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.664054 (+ 222us) service_pool.cc:225] Handling call
1214 20:52:22.072320 (+1408266us) ts_tablet_manager.cc:1151] Deleting tablet 762068d560ff47918dd9fd8791083433
1214 20:52:22.072328 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.122921 (+ 50593us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.161367 (+ 38446us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.193244 (+ 31877us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34558,"tablet-delete.queue_time_us":1385708}
I20251214 20:52:22.196316 1977 tablet_replica.cc:333] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
W20251214 20:52:22.197019 848 leader_election.cc:343] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.197212 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 132) took 1418 ms. Trace:
I20251214 20:52:22.197321 1856 rpcz_store.cc:276] 1214 20:52:20.778851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.778945 (+ 94us) service_pool.cc:225] Handling call
1214 20:52:22.195882 (+1416937us) ts_tablet_manager.cc:1151] Deleting tablet 0d7c7b3f62764ff497f4d70b53eed7de
1214 20:52:22.195893 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.197181 (+ 1288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1416633}
W20251214 20:52:22.197659 989 leader_election.cc:343] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:22.198748 848 leader_election.cc:343] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.198776 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 133) took 1419 ms. Trace:
I20251214 20:52:22.198858 1856 rpcz_store.cc:276] 1214 20:52:20.779019 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.779309 (+ 290us) service_pool.cc:225] Handling call
1214 20:52:22.198652 (+1419343us) ts_tablet_manager.cc:1151] Deleting tablet 1dca369dc5f54a4ea1783e334c5670d6
1214 20:52:22.198662 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.198759 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1419101}
I20251214 20:52:22.199240 909 raft_consensus.cc:2468] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:22.199916 1977 raft_consensus.cc:2243] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.199896 991 leader_election.cc:304] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:22.200222 2038 raft_consensus.cc:2804] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:22.200368 2038 raft_consensus.cc:697] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:22.200357 1977 raft_consensus.cc:2272] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.200635 2038 consensus_queue.cc:237] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.201200 1202 ts_tablet_manager.cc:1434] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251214 20:52:22.202790 1202 tablet_bootstrap.cc:492] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:22.202862 1977 ts_tablet_manager.cc:1916] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.191401 449 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_67fa945ac69a4a599f3dd43cfa93c0ff [id=0d173bfba2b343dbac6a02a5ff84c828] alter complete (version 1)
I20251214 20:52:22.193914 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 762068d560ff47918dd9fd8791083433 (table default.Table_431bf74dfb32400bb0b2ef098c18e8df [id=2fcb81bfc6904fdd924de2494aba2e4a]) successfully deleted
W20251214 20:52:22.208544 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 0d7c7b3f62764ff497f4d70b53eed7de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d7c7b3f62764ff497f4d70b53eed7de already in progress: creating tablet
W20251214 20:52:22.209107 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 1dca369dc5f54a4ea1783e334c5670d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dca369dc5f54a4ea1783e334c5670d6 already in progress: creating tablet
I20251214 20:52:22.214001 1963 raft_consensus.cc:493] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.214188 1963 raft_consensus.cc:515] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:22.214304 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 135) took 1395 ms. Trace:
I20251214 20:52:22.214454 1856 rpcz_store.cc:276] 1214 20:52:20.818910 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.818978 (+ 68us) service_pool.cc:225] Handling call
1214 20:52:22.214084 (+1395106us) ts_tablet_manager.cc:1151] Deleting tablet 16816fa6b8c34207bb929147910b93cb
1214 20:52:22.214102 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.214257 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1394790}
W20251214 20:52:22.214927 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 0d7c7b3f62764ff497f4d70b53eed7de (table default.Table_e26cdfc4ab8f4f19b52de34448bd70e5 [id=756e7314dcb5446f8e62baab11878de2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0d7c7b3f62764ff497f4d70b53eed7de consensus state has no leader
W20251214 20:52:22.215173 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 16816fa6b8c34207bb929147910b93cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16816fa6b8c34207bb929147910b93cb already in progress: creating tablet
I20251214 20:52:22.216892 1963 leader_election.cc:290] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:22.217036 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.216583 1964 raft_consensus.cc:515] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:22.209429 448 rpcz_store.cc:269] 1214 20:52:20.197801 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.228868 (+ 31067us) service_pool.cc:225] Handling call
1214 20:52:20.260543 (+ 31675us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b17aa08 00007ff411c87ca6 00007ff411c879d5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:22.186280 (+1925737us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:20.684056 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.684136 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.684141 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:20.684142 (+ 1us) tablet.cc:660] Decoding operations
1214 20:52:20.684238 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.684246 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.684250 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:20.684326 (+ 76us) tablet.cc:699] Row locks acquired
1214 20:52:20.684328 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:20.684355 (+ 27us) write_op.cc:270] Start()
1214 20:52:20.684377 (+ 22us) write_op.cc:276] Timestamp: P: 1765745540684317 usec, L: 0
1214 20:52:20.684380 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.684515 (+ 135us) log.cc:844] Serialized 901 byte log entry
1214 20:52:20.696503 (+ 11988us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.697694 (+ 1191us) write_op.cc:301] APPLY: starting
1214 20:52:20.697704 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.697732 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.697759 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.697810 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.697834 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:20.697840 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:20.697889 (+ 49us) log.cc:844] Serialized 29 byte log entry
1214 20:52:20.697948 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.697961 (+ 13us) write_op.cc:454] Released schema lock
1214 20:52:20.697966 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:20.698025 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.698028 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
1214 20:52:20.793570 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.793652 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.793656 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:20.793670 (+ 14us) tablet.cc:660] Decoding operations
1214 20:52:20.793741 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.793748 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.793753 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:20.793792 (+ 39us) tablet.cc:699] Row locks acquired
1214 20:52:20.793795 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:20.793820 (+ 25us) write_op.cc:270] Start()
1214 20:52:20.793842 (+ 22us) write_op.cc:276] Timestamp: P: 1765745540793761 usec, L: 0
1214 20:52:20.793846 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.793971 (+ 125us) log.cc:844] Serialized 356 byte log entry
1214 20:52:20.813810 (+ 19839us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.815725 (+ 1915us) write_op.cc:301] APPLY: starting
1214 20:52:20.815735 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.815759 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.815785 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.815825 (+ 40us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.815848 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:20.815854 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:20.815903 (+ 49us) log.cc:844] Serialized 23 byte log entry
1214 20:52:20.815957 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.815969 (+ 12us) write_op.cc:454] Released schema lock
1214 20:52:20.815974 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:20.816037 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.816040 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":62012,"reported_tablets":167,"spinlock_wait_cycles":59935104,"child_traces":[["op",{"apply.queue_time_us":1160,"apply.run_cpu_time_us":325,"apply.run_wall_time_us":325,"num_ops":2,"prepare.queue_time_us":7864,"prepare.run_cpu_time_us":539,"prepare.run_wall_time_us":539,"raft.queue_time_us":239,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":7,"replication_time_us":12108}],["op",{"apply.queue_time_us":1887,"apply.run_cpu_time_us":304,"apply.run_wall_time_us":304,"num_ops":1,"prepare.queue_time_us":16044,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":10927,"raft.queue_time_us":2025,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":19949,"thread_start_us":10394,"threads_started":1}]]}
I20251214 20:52:22.218438 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 137) took 1379 ms. Trace:
I20251214 20:52:22.218531 1856 rpcz_store.cc:276] 1214 20:52:20.839209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.839272 (+ 63us) service_pool.cc:225] Handling call
1214 20:52:22.218289 (+1379017us) ts_tablet_manager.cc:1151] Deleting tablet c8e33ba1149a44c294885d76f676160a
1214 20:52:22.218301 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.218419 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1378724}
I20251214 20:52:22.218930 1964 leader_election.cc:290] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.219307 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.219449 751 tablet_service.cc:1552] Processing DeleteTablet for tablet d6c0f445cd7e44769d79ad5566fd8d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.220156 746 tablet_service.cc:1552] Processing DeleteTablet for tablet f3a2143b5508445492b970d2588cf1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.220425 748 tablet_service.cc:1552] Processing DeleteTablet for tablet d768c45bc02b4d108798efbf3f6aaf7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.220578 751 tablet_service.cc:1552] Processing DeleteTablet for tablet b652a8a80246443491551e76e68d2a8b with delete_type TABLET_DATA_DELETED (replaced by tablet 18e119bc44874aecb4e83c8d6040c8cd) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.220631 749 tablet_service.cc:1552] Processing DeleteTablet for tablet f0d661a1bb034053a4634ff537eafc7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.220803 753 tablet_service.cc:1552] Processing DeleteTablet for tablet b315744c8cd043c5a8bdceec73a7e28b with delete_type TABLET_DATA_DELETED (replaced by tablet f945814952b14131ba4e0c67a403a680) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.220829 744 tablet_service.cc:1552] Processing DeleteTablet for tablet bb698ff30fcc47d09000cc68d2c5225b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:22.220911 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c8e33ba1149a44c294885d76f676160a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8e33ba1149a44c294885d76f676160a already in progress: creating tablet
I20251214 20:52:22.221019 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 324379e3217442c3addcfe58060b9ad7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.221220 1964 raft_consensus.cc:493] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.221380 1964 raft_consensus.cc:515] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.221892 1964 leader_election.cc:290] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.222633 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bbb28259c1442ae94f4eff76c25c539" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:22.222841 1056 raft_consensus.cc:2468] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:22.223295 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b70579ddf96f4499b4e4f235e8f69d67" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:22.223480 1056 raft_consensus.cc:2393] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate a2e8fd1cb731446d88f8e510c7a52999 in this term.
W20251214 20:52:22.223443 1769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20251214 20:52:22.222854 1856 tablet_replica.cc:333] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:22.224167 1856 raft_consensus.cc:2243] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:22.224586 1856 raft_consensus.cc:2272] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:22.225121 712 leader_election.cc:343] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 3bbb28259c1442ae94f4eff76c25c539
I20251214 20:52:22.226074 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b70579ddf96f4499b4e4f235e8f69d67" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:22.227378 712 leader_election.cc:343] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 24ff58f321124d12b5330dca185d426e
I20251214 20:52:22.227774 713 leader_election.cc:304] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:22.228331 1964 raft_consensus.cc:2804] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:22.228425 1964 raft_consensus.cc:493] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:22.228814 1964 raft_consensus.cc:3060] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.232054 1964 raft_consensus.cc:515] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:22.233753 712 leader_election.cc:343] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 3bbb28259c1442ae94f4eff76c25c539
I20251214 20:52:22.234143 1964 leader_election.cc:290] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:22.234355 713 leader_election.cc:343] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 24ff58f321124d12b5330dca185d426e
I20251214 20:52:22.234506 713 leader_election.cc:304] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:22.235049 713 leader_election.cc:304] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:22.235857 1963 raft_consensus.cc:2749] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:22.235725 1202 tablet_bootstrap.cc:654] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.236114 1963 raft_consensus.cc:3060] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.236413 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd6856b800247de896f16eee66f2827 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c3962c33a294fa98f8bd485268aee71) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.236761 744 tablet_service.cc:1552] Processing DeleteTablet for tablet bd8226b431c6456885e928880938969d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.237020 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 8faa7fd7315f485c82aee38bcf4cd8fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.237023 451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 161) took 1157 ms. Trace:
I20251214 20:52:22.237238 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b4f582d8165439d86d9228c146dc41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.237429 744 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b22043d81b493fb12116422b1d3fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet ba7952290e2c4746802cffeb14971da0) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.237145 451 rpcz_store.cc:276] 1214 20:52:21.079643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.994709 (+915066us) service_pool.cc:225] Handling call
1214 20:52:21.995519 (+ 810us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:21.995926 (+ 407us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:22.167832 (+171906us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:22.236829 (+ 68997us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:22.236961 (+ 132us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.115422 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.115546 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.115551 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:22.115554 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.115633 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.115643 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.115647 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:22.115716 (+ 69us) tablet.cc:699] Row locks acquired
1214 20:52:22.115718 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:22.115751 (+ 33us) write_op.cc:270] Start()
1214 20:52:22.115780 (+ 29us) write_op.cc:276] Timestamp: P: 1765745542115733 usec, L: 0
1214 20:52:22.115783 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.117367 (+ 1584us) log.cc:844] Serialized 433 byte log entry
1214 20:52:22.166804 (+ 49437us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.166953 (+ 149us) write_op.cc:301] APPLY: starting
1214 20:52:22.166972 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.167019 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.167049 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.167195 (+ 146us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.167232 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:22.167239 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.167326 (+ 87us) log.cc:844] Serialized 29 byte log entry
1214 20:52:22.167397 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.167418 (+ 21us) write_op.cc:454] Released schema lock
1214 20:52:22.167426 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.167502 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.167506 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":93,"apply.run_cpu_time_us":542,"apply.run_wall_time_us":542,"num_ops":2,"prepare.queue_time_us":119070,"prepare.run_cpu_time_us":654,"prepare.run_wall_time_us":4429,"raft.queue_time_us":312,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":7,"replication_time_us":50984}]]}
I20251214 20:52:22.237847 1982 ts_tablet_manager.cc:1950] T b1df0dad2fb040a5a12e0401e00c87db P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:22.237885 451 master_service.cc:954] ReplaceTablet: received request to replace tablet fe24e4f4e74040249068505058c15622 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:22.238332 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bbb28259c1442ae94f4eff76c25c539" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:22.238525 1056 raft_consensus.cc:3060] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.238890 1963 raft_consensus.cc:2749] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:22.239013 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 104) took 2017 ms. Trace:
I20251214 20:52:22.239173 1982 rpcz_store.cc:276] 1214 20:52:20.221183 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.221233 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:22.104233 (+1883000us) ts_tablet_manager.cc:1151] Deleting tablet b1df0dad2fb040a5a12e0401e00c87db
1214 20:52:22.104241 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.113897 (+ 9656us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.145368 (+ 31471us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.238962 (+ 93594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1731613}
I20251214 20:52:22.239506 753 tablet_service.cc:1552] Processing DeleteTablet for tablet c25832da5c1644329e18653000f3330e with delete_type TABLET_DATA_DELETED (replaced by tablet 0c26d1f969ed4071a2f2cf1615a17cac) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.239770 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 105) took 2016 ms. Trace:
I20251214 20:52:22.239915 1982 rpcz_store.cc:276] 1214 20:52:20.223611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.223674 (+ 63us) service_pool.cc:225] Handling call
1214 20:52:22.239589 (+2015915us) ts_tablet_manager.cc:1151] Deleting tablet 65e79db74fa046698507c83416ee1669
1214 20:52:22.239601 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.239735 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1866907}
I20251214 20:52:22.240237 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 106) took 2016 ms. Trace:
I20251214 20:52:22.240345 1982 rpcz_store.cc:276] 1214 20:52:20.223928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.224147 (+ 219us) service_pool.cc:225] Handling call
1214 20:52:22.240133 (+2015986us) ts_tablet_manager.cc:1151] Deleting tablet 8d74d3abc45a41969289c0a0892bfa04
1214 20:52:22.240141 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.240208 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1866728}
I20251214 20:52:22.240624 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 107) took 2016 ms. Trace:
I20251214 20:52:22.240728 1982 rpcz_store.cc:276] 1214 20:52:20.224379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.224427 (+ 48us) service_pool.cc:225] Handling call
1214 20:52:22.240514 (+2016087us) ts_tablet_manager.cc:1151] Deleting tablet 012b81223d0f4ae9b742014e329f11a9
1214 20:52:22.240521 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.240595 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1867044}
I20251214 20:52:22.240980 1982 tablet_replica.cc:333] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:22.241109 1056 raft_consensus.cc:2468] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:22.241205 1982 raft_consensus.cc:2243] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:22.241678 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65b2fe0de22646e4b77fbc6a289ff4a1 consensus state has no leader
I20251214 20:52:22.241881 450 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 164) took 1069 ms. Trace:
W20251214 20:52:22.241974 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9da5e98c6f4843449886176e9fcfa8e1 consensus state has no leader
I20251214 20:52:22.242096 453 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 163) took 1070 ms. Trace:
I20251214 20:52:22.242195 453 rpcz_store.cc:276] 1214 20:52:21.171880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.010308 (+838428us) service_pool.cc:225] Handling call
1214 20:52:22.061397 (+ 51089us) spinlock_profiling.cc:243] Waited 50.8 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b182456 00007ff411c8e146 00007ff411c8de75 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:22.061609 (+ 212us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:22.061990 (+ 381us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:22.062174 (+ 184us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:22.062263 (+ 89us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:22.191501 (+129238us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:22.242065 (+ 50564us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.134411 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.134518 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.134525 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:22.134527 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:22.134632 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.134643 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.134647 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.134724 (+ 77us) tablet.cc:699] Row locks acquired
1214 20:52:22.134727 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.134766 (+ 39us) write_op.cc:270] Start()
1214 20:52:22.134801 (+ 35us) write_op.cc:276] Timestamp: P: 1765745542134742 usec, L: 0
1214 20:52:22.134805 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.135000 (+ 195us) log.cc:844] Serialized 405 byte log entry
1214 20:52:22.190218 (+ 55218us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.190910 (+ 692us) write_op.cc:301] APPLY: starting
1214 20:52:22.190920 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.190955 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.190983 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.191016 (+ 33us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.191039 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.191045 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:22.191093 (+ 48us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.191157 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.191169 (+ 12us) write_op.cc:454] Released schema lock
1214 20:52:22.191173 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":116838144,"child_traces":[["op",{"apply.queue_time_us":661,"apply.run_cpu_time_us":316,"apply.run_wall_time_us":316,"num_ops":1,"prepare.queue_time_us":71744,"prepare.run_cpu_time_us":699,"prepare.run_wall_time_us":1109,"raft.queue_time_us":36,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":55400,"threads_started":1}]]}
I20251214 20:52:22.242719 713 leader_election.cc:304] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:22.241994 450 rpcz_store.cc:276] 1214 20:52:21.172018 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.013534 (+841516us) service_pool.cc:225] Handling call
1214 20:52:22.059892 (+ 46358us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b182456 00007ff411c8e146 00007ff411c8de75 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:22.060140 (+ 248us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:22.060359 (+ 219us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:22.060555 (+ 196us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:22.060661 (+ 106us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:22.191601 (+130940us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:22.241845 (+ 50244us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.126301 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.126468 (+ 167us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.126474 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:22.126476 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:22.126611 (+ 135us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.126622 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.126626 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.126695 (+ 69us) tablet.cc:699] Row locks acquired
1214 20:52:22.126697 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:22.126876 (+ 179us) write_op.cc:270] Start()
1214 20:52:22.126924 (+ 48us) write_op.cc:276] Timestamp: P: 1765745542126714 usec, L: 0
1214 20:52:22.126928 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.127167 (+ 239us) log.cc:844] Serialized 405 byte log entry
1214 20:52:22.190162 (+ 62995us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.190508 (+ 346us) write_op.cc:301] APPLY: starting
1214 20:52:22.190517 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.190538 (+ 21us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.190562 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.190587 (+ 25us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.190605 (+ 18us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.190610 (+ 5us) write_op.cc:312] APPLY: finished
1214 20:52:22.190659 (+ 49us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.191459 (+ 800us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.191470 (+ 11us) write_op.cc:454] Released schema lock
1214 20:52:22.191476 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.191527 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.191530 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":105898368,"child_traces":[["op",{"apply.queue_time_us":291,"apply.run_cpu_time_us":286,"apply.run_wall_time_us":1015,"mutex_wait_us":739,"num_ops":1,"prepare.queue_time_us":65074,"prepare.run_cpu_time_us":1035,"prepare.run_wall_time_us":8082,"raft.queue_time_us":2247,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":63219}]]}
I20251214 20:52:22.244223 450 master_service.cc:954] ReplaceTablet: received request to replace tablet 02e256d4537249a394b80dc36beefef4 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:22.244585 1856 ts_tablet_manager.cc:1916] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.245153 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet b1df0dad2fb040a5a12e0401e00c87db (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]) successfully deleted
W20251214 20:52:22.245457 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 65e79db74fa046698507c83416ee1669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e79db74fa046698507c83416ee1669 already in progress: creating tablet
W20251214 20:52:22.245815 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8d74d3abc45a41969289c0a0892bfa04 because the tablet was not found. No further retry: Not found: Tablet not found: 8d74d3abc45a41969289c0a0892bfa04
I20251214 20:52:22.245971 1102 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
W20251214 20:52:22.246201 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 012b81223d0f4ae9b742014e329f11a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012b81223d0f4ae9b742014e329f11a9 already in progress: creating tablet
W20251214 20:52:22.246215 1769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 20:52:22.246409 1769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251214 20:52:22.246771 1963 raft_consensus.cc:2804] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:22.247000 1963 raft_consensus.cc:697] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:22.247304 1963 consensus_queue.cc:237] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:22.247890 1982 raft_consensus.cc:2272] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.248651 1982 ts_tablet_manager.cc:1916] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.255950 1206 tablet_bootstrap.cc:492] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:22.256167 1206 ts_tablet_manager.cc:1403] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.130s user 0.005s sys 0.009s
I20251214 20:52:22.257575 1206 raft_consensus.cc:359] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.265614 753 tablet_service.cc:1552] Processing DeleteTablet for tablet a063083f54ad49d2a842fa080d44ed85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.265772 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 63497e77baeb4d328be2421b5732a28a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.266331 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 2da66ec83e9140f7a605f0c86abff468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.267026 744 tablet_service.cc:1552] Processing DeleteTablet for tablet ca263a80702f4f64aa7c872c2be08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.267292 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 40a9a6192c744282ab5672cbe15fd299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.270697 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.271188 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.273051 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 06381efe75284e649fd6d1a9903cfa9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.273187 744 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.273568 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.273852 744 tablet_service.cc:1552] Processing DeleteTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f with delete_type TABLET_DATA_DELETED (replaced by tablet f11834cabdad445b8e53cd237d146282) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.274117 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 292a28b12df246a8bc855d81c530746e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.274247 1977 ts_tablet_manager.cc:1929] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.274427 1977 log.cc:1199] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/2da66ec83e9140f7a605f0c86abff468
I20251214 20:52:22.274653 744 tablet_service.cc:1552] Processing DeleteTablet for tablet ee7d709316b24b8eb223a838710c2979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.274758 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dd657f3baff40bb8cb7f4648367f7a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.274855 1977 ts_tablet_manager.cc:1950] T 2da66ec83e9140f7a605f0c86abff468 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:22.275740 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 109) took 1439 ms. Trace:
I20251214 20:52:22.276021 1202 tablet_bootstrap.cc:492] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:22.276247 1202 ts_tablet_manager.cc:1403] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.074s user 0.006s sys 0.000s
I20251214 20:52:22.275847 1977 rpcz_store.cc:276] 1214 20:52:20.836442 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.837678 (+ 1236us) service_pool.cc:225] Handling call
1214 20:52:22.176405 (+1338727us) ts_tablet_manager.cc:1151] Deleting tablet 2da66ec83e9140f7a605f0c86abff468
1214 20:52:22.176439 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.258639 (+ 82200us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.274104 (+ 15465us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.275666 (+ 1562us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1274685}
I20251214 20:52:22.277294 1202 raft_consensus.cc:359] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.277523 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 2da66ec83e9140f7a605f0c86abff468 (table default.Table_2325a2c5e97f45ebac4c943380bb3525 [id=0c6d9651f90f479293852cb3fb1d320f]) successfully deleted
I20251214 20:52:22.278108 1982 ts_tablet_manager.cc:1929] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.280002 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_f2360eda2fac4147840743a9a4f0bc2b" } new_table_name: "default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3" modify_external_catalogs: true
I20251214 20:52:22.284396 1856 ts_tablet_manager.cc:1929] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.284448 1202 raft_consensus.cc:385] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.284574 1856 log.cc:1199] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/0cafed2861e94cd9b0d10d33864ba45d
I20251214 20:52:22.284574 1202 raft_consensus.cc:740] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.284493 1977 tablet_replica.cc:333] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:22.285240 1202 consensus_queue.cc:260] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.285974 1202 ts_tablet_manager.cc:1434] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251214 20:52:22.286317 1202 tablet_bootstrap.cc:492] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:22.286890 1977 raft_consensus.cc:2243] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.287384 1977 raft_consensus.cc:2272] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.288266 1977 ts_tablet_manager.cc:1916] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.289274 1202 tablet_bootstrap.cc:654] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.293409 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet dfe50a6219134431b0a025317932609b
I20251214 20:52:22.293905 447 catalog_manager.cc:6724] ReplaceTablet: tablet dfe50a6219134431b0a025317932609b deleted and replaced by tablet 6b482043af42485eb4c73d6eb864a963
I20251214 20:52:22.294026 447 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 162) took 1127 ms. Trace:
I20251214 20:52:22.294117 447 rpcz_store.cc:276] 1214 20:52:21.166292 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.006322 (+840030us) service_pool.cc:225] Handling call
1214 20:52:22.293997 (+287675us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.135540 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.135635 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.135641 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:22.135644 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.135758 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.135768 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.135773 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:22.135842 (+ 69us) tablet.cc:699] Row locks acquired
1214 20:52:22.135844 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:22.135877 (+ 33us) write_op.cc:270] Start()
1214 20:52:22.135905 (+ 28us) write_op.cc:276] Timestamp: P: 1765745542135855 usec, L: 0
1214 20:52:22.135908 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.136070 (+ 162us) log.cc:844] Serialized 622 byte log entry
1214 20:52:22.275400 (+139330us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.286969 (+ 11569us) write_op.cc:301] APPLY: starting
1214 20:52:22.286989 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.287038 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.287088 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.287268 (+ 180us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.287325 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.287332 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.287429 (+ 97us) log.cc:844] Serialized 29 byte log entry
1214 20:52:22.287538 (+ 109us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.287561 (+ 23us) write_op.cc:454] Released schema lock
1214 20:52:22.287569 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":11475,"num_ops":2,"prepare.queue_time_us":26395,"prepare.run_cpu_time_us":659,"prepare.run_wall_time_us":2962,"raft.queue_time_us":45,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":139459}]]}
I20251214 20:52:22.297327 1977 ts_tablet_manager.cc:1929] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.297462 1977 log.cc:1199] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/0cafed2861e94cd9b0d10d33864ba45d
I20251214 20:52:22.297863 1977 ts_tablet_manager.cc:1950] T 0cafed2861e94cd9b0d10d33864ba45d P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:22.297922 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet dfe50a6219134431b0a025317932609b with delete_type TABLET_DATA_DELETED (replaced by tablet 6b482043af42485eb4c73d6eb864a963) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.298687 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 118) took 1398 ms. Trace:
I20251214 20:52:22.298838 1977 rpcz_store.cc:276] 1214 20:52:20.899694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.899738 (+ 44us) service_pool.cc:225] Handling call
1214 20:52:22.276697 (+1376959us) ts_tablet_manager.cc:1151] Deleting tablet 0cafed2861e94cd9b0d10d33864ba45d
1214 20:52:22.276706 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.294105 (+ 17399us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.297183 (+ 3078us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.298624 (+ 1441us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1037,"tablet-delete.queue_time_us":1374858}
I20251214 20:52:22.299412 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 0cafed2861e94cd9b0d10d33864ba45d (table default.Table_ba612520b0be474db6573ce754044c8a [id=1bf28f656cfe496d8accd428d0b238f2]) successfully deleted
I20251214 20:52:22.299638 895 tablet_service.cc:1552] Processing DeleteTablet for tablet dfe50a6219134431b0a025317932609b with delete_type TABLET_DATA_DELETED (replaced by tablet 6b482043af42485eb4c73d6eb864a963) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.301904 1856 ts_tablet_manager.cc:1950] T 0cafed2861e94cd9b0d10d33864ba45d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:22.300282 753 tablet_service.cc:1552] Processing DeleteTablet for tablet dfe50a6219134431b0a025317932609b with delete_type TABLET_DATA_DELETED (replaced by tablet 6b482043af42485eb4c73d6eb864a963) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.303442 1982 log.cc:1199] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/4b86f4adaf4540f48a4f42bc45e259e5
I20251214 20:52:22.304546 1202 tablet_bootstrap.cc:492] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:22.304738 1202 ts_tablet_manager.cc:1403] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.001s
I20251214 20:52:22.305056 1982 ts_tablet_manager.cc:1950] T 4b86f4adaf4540f48a4f42bc45e259e5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:22.305718 1202 raft_consensus.cc:359] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.305938 1202 raft_consensus.cc:385] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.306033 1202 raft_consensus.cc:740] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.306114 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 138) took 1465 ms. Trace:
I20251214 20:52:22.306238 1856 rpcz_store.cc:276] 1214 20:52:20.841009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.841065 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:22.222782 (+1381717us) ts_tablet_manager.cc:1151] Deleting tablet 0cafed2861e94cd9b0d10d33864ba45d
1214 20:52:22.222805 (+ 23us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.252068 (+ 29263us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.284216 (+ 32148us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.306055 (+ 21839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1381467}
I20251214 20:52:22.306331 1202 consensus_queue.cc:260] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.307821 1202 ts_tablet_manager.cc:1434] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:22.312219 1202 tablet_bootstrap.cc:492] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:22.313068 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 117) took 1413 ms. Trace:
I20251214 20:52:22.313267 1977 rpcz_store.cc:276] 1214 20:52:20.899518 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.899566 (+ 48us) service_pool.cc:225] Handling call
1214 20:52:22.308731 (+1409165us) ts_tablet_manager.cc:1151] Deleting tablet c8e33ba1149a44c294885d76f676160a
1214 20:52:22.312813 (+ 4082us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.313000 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406801}
W20251214 20:52:22.313896 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c8e33ba1149a44c294885d76f676160a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8e33ba1149a44c294885d76f676160a already in progress: creating tablet
I20251214 20:52:22.314277 1977 tablet_replica.cc:333] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:22.314306 1202 tablet_bootstrap.cc:654] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.316483 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet fe24e4f4e74040249068505058c15622
I20251214 20:52:22.317020 451 catalog_manager.cc:6724] ReplaceTablet: tablet fe24e4f4e74040249068505058c15622 deleted and replaced by tablet 0dbfe8ee68bf4a2c88489b1f81b36991
I20251214 20:52:22.317160 451 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 165) took 1138 ms. Trace:
I20251214 20:52:22.317274 451 rpcz_store.cc:276] 1214 20:52:21.179166 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.237807 (+1058641us) service_pool.cc:225] Handling call
1214 20:52:22.278457 (+ 40650us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b18cf0b 00007ff411c97107 00007ff411c96e35 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:22.317121 (+ 38664us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.307013 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.307192 (+ 179us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.307198 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:22.307200 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:22.307346 (+ 146us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.307362 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.307368 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:22.307455 (+ 87us) tablet.cc:699] Row locks acquired
1214 20:52:22.307458 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.307495 (+ 37us) write_op.cc:270] Start()
1214 20:52:22.307531 (+ 36us) write_op.cc:276] Timestamp: P: 1765745542307436 usec, L: 0
1214 20:52:22.307536 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.307760 (+ 224us) log.cc:844] Serialized 622 byte log entry
1214 20:52:22.311643 (+ 3883us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.315474 (+ 3831us) write_op.cc:301] APPLY: starting
1214 20:52:22.315491 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.315532 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.315571 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.315684 (+ 113us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.315744 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.315752 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:22.315827 (+ 75us) log.cc:844] Serialized 29 byte log entry
1214 20:52:22.316153 (+ 326us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.316167 (+ 14us) write_op.cc:454] Released schema lock
1214 20:52:22.316172 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.316228 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.316232 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":92682496,"child_traces":[["op",{"apply.queue_time_us":3773,"apply.run_cpu_time_us":484,"apply.run_wall_time_us":751,"num_ops":2,"prepare.queue_time_us":27636,"prepare.run_cpu_time_us":908,"prepare.run_wall_time_us":1795,"raft.queue_time_us":77,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":777,"replication_time_us":4075}]]}
I20251214 20:52:22.318274 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_3024ac839d9441db89239acebb00bc12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.318863 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3024ac839d9441db89239acebb00bc12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.318850 1977 raft_consensus.cc:2243] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.319092 1977 raft_consensus.cc:2272] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:22.319633 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:39607 (request call id 152) took 2409 ms (client timeout 1000 ms). Trace:
I20251214 20:52:22.319861 1977 ts_tablet_manager.cc:1916] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:22.319749 452 rpcz_store.cc:269] 1214 20:52:19.910025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:19.914677 (+ 4652us) service_pool.cc:225] Handling call
1214 20:52:22.319561 (+2404884us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:20.546291 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:20.546453 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:20.546462 (+ 9us) write_op.cc:435] Acquired schema lock
1214 20:52:20.546464 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:20.546590 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:20.546607 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:20.546613 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:20.546701 (+ 88us) tablet.cc:699] Row locks acquired
1214 20:52:20.546703 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:20.546751 (+ 48us) write_op.cc:270] Start()
1214 20:52:20.546793 (+ 42us) write_op.cc:276] Timestamp: P: 1765745540546608 usec, L: 0
1214 20:52:20.546796 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:20.547026 (+ 230us) log.cc:844] Serialized 356 byte log entry
1214 20:52:20.550668 (+ 3642us) op_driver.cc:464] REPLICATION: finished
1214 20:52:20.552058 (+ 1390us) write_op.cc:301] APPLY: starting
1214 20:52:20.552084 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:20.552132 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:20.552160 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:20.552226 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:20.552260 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:20.552267 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:20.552356 (+ 89us) log.cc:844] Serialized 23 byte log entry
1214 20:52:20.552419 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.552452 (+ 33us) write_op.cc:454] Released schema lock
1214 20:52:20.552458 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:20.552537 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:20.552541 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":351,"reported_tablets":164,"child_traces":[["op",{"apply.queue_time_us":1301,"apply.run_cpu_time_us":484,"apply.run_wall_time_us":484,"num_ops":1,"prepare.queue_time_us":288,"prepare.run_cpu_time_us":909,"prepare.run_wall_time_us":909,"raft.queue_time_us":489,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":283,"replication_time_us":3844}]]}
I20251214 20:52:22.320537 452 master_service.cc:954] ReplaceTablet: received request to replace tablet 6eca34627bbc4a3f9915be72f3128370 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:22.323096 1856 tablet_replica.cc:333] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:22.323421 1856 raft_consensus.cc:2243] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.323601 1856 raft_consensus.cc:2272] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.324172 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 0cafed2861e94cd9b0d10d33864ba45d (table default.Table_ba612520b0be474db6573ce754044c8a [id=1bf28f656cfe496d8accd428d0b238f2]) successfully deleted
I20251214 20:52:22.324271 1856 ts_tablet_manager.cc:1916] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.324705 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 109) took 2099 ms. Trace:
I20251214 20:52:22.324828 1982 rpcz_store.cc:276] 1214 20:52:20.225297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.225340 (+ 43us) service_pool.cc:225] Handling call
1214 20:52:22.240914 (+2015574us) ts_tablet_manager.cc:1151] Deleting tablet 4b86f4adaf4540f48a4f42bc45e259e5
1214 20:52:22.240922 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.274836 (+ 33914us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.277978 (+ 3142us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.324656 (+ 46678us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1867391}
I20251214 20:52:22.325431 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24e4f4e74040249068505058c15622 with delete_type TABLET_DATA_DELETED (replaced by tablet 0dbfe8ee68bf4a2c88489b1f81b36991) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.326830 1982 tablet_replica.cc:333] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:22.327054 1982 raft_consensus.cc:2243] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.327235 1982 raft_consensus.cc:2272] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.327548 895 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24e4f4e74040249068505058c15622 with delete_type TABLET_DATA_DELETED (replaced by tablet 0dbfe8ee68bf4a2c88489b1f81b36991) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.327818 1982 ts_tablet_manager.cc:1916] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.328475 1203 tablet_bootstrap.cc:492] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:22.328650 1203 ts_tablet_manager.cc:1403] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.148s user 0.020s sys 0.019s
I20251214 20:52:22.328763 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 242a77ae57174de48f4eaf43384ee0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.328995 749 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24e4f4e74040249068505058c15622 with delete_type TABLET_DATA_DELETED (replaced by tablet 0dbfe8ee68bf4a2c88489b1f81b36991) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.329087 753 tablet_service.cc:1552] Processing DeleteTablet for tablet ba50e32f7e724868bb891045a939e362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.329217 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 65e79db74fa046698507c83416ee1669 with delete_type TABLET_DATA_DELETED (replaced by tablet 4dcd511a5dca4a2cbda81fbc255ce354) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.329383 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 35a529702f924a5c9bf53c3c668c6094 with delete_type TABLET_DATA_DELETED (replaced by tablet 719d157ac8f74eb7911df05ab5e4a3d5) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.329528 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7c6fe3b24daca70711a1990af556 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.329510 1203 raft_consensus.cc:359] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.329555 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 1dca369dc5f54a4ea1783e334c5670d6) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.329679 748 tablet_service.cc:1552] Processing DeleteTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.329707 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 69f6108792fa4c2a9744e3248c76251f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.329712 1203 raft_consensus.cc:385] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.329772 1203 raft_consensus.cc:740] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.329833 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d89ee602c524c96a21c0a235af07481 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e11723c042a4637839f1b13bed0935a) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.329982 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d6e7306254044449f1127fb963ca5f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.329972 2038 raft_consensus.cc:493] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.330159 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 4b86f4adaf4540f48a4f42bc45e259e5 (table default.Table_69c0719fe94a4c7594516a739f6a40ea [id=a5424dd4eee64d0686d5e4a8a5c35213]) successfully deleted
I20251214 20:52:22.330125 2038 raft_consensus.cc:515] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.330890 2038 leader_election.cc:290] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.331491 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65b2fe0de22646e4b77fbc6a289ff4a1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:22.331640 744 tablet_service.cc:1552] Processing DeleteTablet for tablet b1df0dad2fb040a5a12e0401e00c87db with delete_type TABLET_DATA_DELETED (replaced by tablet 8fcdd6bafa5c4aa6b59668cde523d010) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.331760 909 raft_consensus.cc:2468] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:22.331807 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b48bb260e854beb9ec61ce9c79b46c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 566c6e4761d5486d964e276d151e32b0) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.331957 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 012b81223d0f4ae9b742014e329f11a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.332088 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.332098 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65b2fe0de22646e4b77fbc6a289ff4a1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:22.329990 1203 consensus_queue.cc:260] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.332247 991 leader_election.cc:304] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.332669 2079 raft_consensus.cc:2804] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251214 20:52:22.332707 989 leader_election.cc:343] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.332762 2079 raft_consensus.cc:493] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:22.332831 2079 raft_consensus.cc:3060] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.334815 1963 raft_consensus.cc:493] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.334975 1963 raft_consensus.cc:515] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:22.336539 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f4760585baf4b4ab3b9b94b5a748a61" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:22.337385 1963 leader_election.cc:290] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:22.337877 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f4760585baf4b4ab3b9b94b5a748a61" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:22.338821 713 leader_election.cc:304] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:22.332686 1203 ts_tablet_manager.cc:1434] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:22.339262 1203 tablet_bootstrap.cc:492] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:22.340952 1963 raft_consensus.cc:2749] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:22.340384 2079 raft_consensus.cc:515] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.344621 2079 leader_election.cc:290] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.345093 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65b2fe0de22646e4b77fbc6a289ff4a1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:22.345223 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65b2fe0de22646e4b77fbc6a289ff4a1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:22.345436 909 raft_consensus.cc:3060] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:22.345791 989 leader_election.cc:343] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.348012 1977 ts_tablet_manager.cc:1929] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:22.348189 1977 log.cc:1199] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/3c3962c33a294fa98f8bd485268aee71
I20251214 20:52:22.348599 1977 ts_tablet_manager.cc:1950] T 3c3962c33a294fa98f8bd485268aee71 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:22.348493 1762 consensus_queue.cc:1048] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:22.349902 909 raft_consensus.cc:2468] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:22.350445 991 leader_election.cc:304] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:22.350756 2079 raft_consensus.cc:2804] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:22.351529 1203 tablet_bootstrap.cc:654] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.352798 1762 consensus_queue.cc:1048] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:22.352979 2079 raft_consensus.cc:697] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:22.353246 2079 consensus_queue.cc:237] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.354357 451 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 168) took 1146 ms. Trace:
I20251214 20:52:22.354523 1762 raft_consensus.cc:493] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.354660 1762 raft_consensus.cc:515] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.354494 451 rpcz_store.cc:276] 1214 20:52:21.208128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.318039 (+1109911us) service_pool.cc:225] Handling call
1214 20:52:22.318974 (+ 935us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:22.331095 (+ 12121us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:22.353832 (+ 22737us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:22.353960 (+ 128us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:22.354314 (+ 354us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.340601 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.340734 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.340742 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:22.340744 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:22.340857 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.340873 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.340878 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:22.341002 (+ 124us) tablet.cc:699] Row locks acquired
1214 20:52:22.341005 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.341055 (+ 50us) write_op.cc:270] Start()
1214 20:52:22.341097 (+ 42us) write_op.cc:276] Timestamp: P: 1765745542340986 usec, L: 0
1214 20:52:22.341101 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.341343 (+ 242us) log.cc:844] Serialized 433 byte log entry
1214 20:52:22.352451 (+ 11108us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.353353 (+ 902us) write_op.cc:301] APPLY: starting
1214 20:52:22.353370 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.353413 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.353438 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.353550 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.353586 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:22.353593 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.353665 (+ 72us) log.cc:844] Serialized 29 byte log entry
1214 20:52:22.353728 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.353747 (+ 19us) write_op.cc:454] Released schema lock
1214 20:52:22.353754 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":840,"num_ops":2,"prepare.queue_time_us":8873,"prepare.run_cpu_time_us":894,"prepare.run_wall_time_us":2506,"raft.queue_time_us":1586,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":11319}]]}
I20251214 20:52:22.355319 1762 leader_election.cc:290] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.355813 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8ca3b0a5a22a425b866b0b2565f80fac" } new_table_name: "default.Table_4e0d19a09c1f44eb85c4a776f6cd599a" modify_external_catalogs: true
I20251214 20:52:22.355850 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac64f4fcb6a943668584658df19f950c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:22.356739 848 leader_election.cc:343] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.355842 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac64f4fcb6a943668584658df19f950c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:22.357079 1054 raft_consensus.cc:2468] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:22.357615 851 leader_election.cc:304] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:22.358107 1650 raft_consensus.cc:2804] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:22.358227 1650 raft_consensus.cc:493] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:22.358315 1650 raft_consensus.cc:3060] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.361018 1650 raft_consensus.cc:515] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.361698 1650 leader_election.cc:290] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.365638 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac64f4fcb6a943668584658df19f950c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:22.365859 1054 raft_consensus.cc:3060] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.366600 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 5157ef5f1fd541efbbd2887bf4505254 (DEFAULT_TABLE table=default.Table_d80c747f5a704a608b5e28b5e5d247b1 [id=e09da957fd5b4879a87ffb8c1cdd4bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.366859 746 tablet_service.cc:1505] Processing CreateTablet for tablet 5157ef5f1fd541efbbd2887bf4505254 (DEFAULT_TABLE table=default.Table_d80c747f5a704a608b5e28b5e5d247b1 [id=e09da957fd5b4879a87ffb8c1cdd4bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.367100 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5157ef5f1fd541efbbd2887bf4505254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.367271 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5157ef5f1fd541efbbd2887bf4505254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.367390 891 tablet_service.cc:1505] Processing CreateTablet for tablet 5157ef5f1fd541efbbd2887bf4505254 (DEFAULT_TABLE table=default.Table_d80c747f5a704a608b5e28b5e5d247b1 [id=e09da957fd5b4879a87ffb8c1cdd4bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.367805 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5157ef5f1fd541efbbd2887bf4505254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.369065 1054 raft_consensus.cc:2468] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:22.369699 851 leader_election.cc:304] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.369995 1762 raft_consensus.cc:2804] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:22.370095 1762 raft_consensus.cc:697] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:22.370318 1762 consensus_queue.cc:237] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.371969 1964 raft_consensus.cc:493] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.372126 1964 raft_consensus.cc:515] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:22.372733 1964 leader_election.cc:290] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:22.375590 713 leader_election.cc:343] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 3c3962c33a294fa98f8bd485268aee71
I20251214 20:52:22.377553 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3c3962c33a294fa98f8bd485268aee71" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:22.378237 712 leader_election.cc:343] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: Tablet not RUNNING: STOPPED
I20251214 20:52:22.378360 712 leader_election.cc:304] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:22.378816 1964 raft_consensus.cc:2749] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:22.385926 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 02e256d4537249a394b80dc36beefef4
I20251214 20:52:22.386430 450 catalog_manager.cc:6724] ReplaceTablet: tablet 02e256d4537249a394b80dc36beefef4 deleted and replaced by tablet cea9423c40044e4cabb8bbd755a46688
I20251214 20:52:22.386574 450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 167) took 1180 ms. Trace:
I20251214 20:52:22.386667 450 rpcz_store.cc:276] 1214 20:52:21.205933 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.244169 (+1038236us) service_pool.cc:225] Handling call
1214 20:52:22.278231 (+ 34062us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b18cf0b 00007ff411c97107 00007ff411c96e35 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:22.386522 (+108291us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.379659 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.379811 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.379817 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:22.379820 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.379949 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.379961 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.379966 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:22.380047 (+ 81us) tablet.cc:699] Row locks acquired
1214 20:52:22.380050 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.380076 (+ 26us) write_op.cc:270] Start()
1214 20:52:22.380107 (+ 31us) write_op.cc:276] Timestamp: P: 1765745542380006 usec, L: 0
1214 20:52:22.380119 (+ 12us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.380295 (+ 176us) log.cc:844] Serialized 622 byte log entry
1214 20:52:22.383094 (+ 2799us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.384504 (+ 1410us) write_op.cc:301] APPLY: starting
1214 20:52:22.384518 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.384554 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.384587 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.384687 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.384742 (+ 55us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.384750 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:22.384822 (+ 72us) log.cc:844] Serialized 29 byte log entry
1214 20:52:22.384873 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.385647 (+ 774us) write_op.cc:454] Released schema lock
1214 20:52:22.385655 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":77597952,"child_traces":[["op",{"apply.queue_time_us":1362,"mutex_wait_us":557,"num_ops":2,"prepare.queue_time_us":15248,"raft.queue_time_us":602,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":1932,"replication_time_us":2947}]]}
I20251214 20:52:22.391779 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e256d4537249a394b80dc36beefef4 with delete_type TABLET_DATA_DELETED (replaced by tablet cea9423c40044e4cabb8bbd755a46688) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.392771 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e256d4537249a394b80dc36beefef4 with delete_type TABLET_DATA_DELETED (replaced by tablet cea9423c40044e4cabb8bbd755a46688) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.395375 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e256d4537249a394b80dc36beefef4 with delete_type TABLET_DATA_DELETED (replaced by tablet cea9423c40044e4cabb8bbd755a46688) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.399847 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ebccdf7648384d44b9629a1b41509e68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.400573 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebccdf7648384d44b9629a1b41509e68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.408735 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 171) took 1177 ms. Trace:
I20251214 20:52:22.408883 450 rpcz_store.cc:276] 1214 20:52:21.231273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.387299 (+1156026us) service_pool.cc:225] Handling call
1214 20:52:22.399755 (+ 12456us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b1801c3 00007ff411c8a7c6 00007ff411c8a4f5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:22.400687 (+ 932us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:22.401128 (+ 441us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:22.408313 (+ 7185us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:22.408558 (+ 245us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:22.408662 (+ 104us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.401958 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.402071 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.402075 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:22.402078 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.402183 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.402196 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.402201 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:22.402275 (+ 74us) tablet.cc:699] Row locks acquired
1214 20:52:22.402278 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.402317 (+ 39us) write_op.cc:270] Start()
1214 20:52:22.402351 (+ 34us) write_op.cc:276] Timestamp: P: 1765745542402239 usec, L: 0
1214 20:52:22.402355 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.402549 (+ 194us) log.cc:844] Serialized 433 byte log entry
1214 20:52:22.405557 (+ 3008us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.407840 (+ 2283us) write_op.cc:301] APPLY: starting
1214 20:52:22.407855 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.407893 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.407937 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.408045 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.408078 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:22.408085 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.408162 (+ 77us) log.cc:844] Serialized 29 byte log entry
1214 20:52:22.408213 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.408231 (+ 18us) write_op.cc:454] Released schema lock
1214 20:52:22.408237 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":27877632,"child_traces":[["op",{"apply.queue_time_us":2229,"num_ops":2,"prepare.queue_time_us":178,"raft.queue_time_us":34,"replication_time_us":3175}]]}
I20251214 20:52:22.412398 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 1e185d107619414a8bcc15ad496547dc (DEFAULT_TABLE table=default.Table_5577714724264f51932f490a337aa8d2 [id=c3e26e52ceef4e29a1ae13efb3d380d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.412992 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e185d107619414a8bcc15ad496547dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.416442 746 tablet_service.cc:1505] Processing CreateTablet for tablet 1e185d107619414a8bcc15ad496547dc (DEFAULT_TABLE table=default.Table_5577714724264f51932f490a337aa8d2 [id=c3e26e52ceef4e29a1ae13efb3d380d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.416395 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_0fc25951e7cc4aebac50ad75316773f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:22.416953 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e185d107619414a8bcc15ad496547dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:22.417074 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fc25951e7cc4aebac50ad75316773f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.420152 891 tablet_service.cc:1505] Processing CreateTablet for tablet 1e185d107619414a8bcc15ad496547dc (DEFAULT_TABLE table=default.Table_5577714724264f51932f490a337aa8d2 [id=c3e26e52ceef4e29a1ae13efb3d380d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.420627 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e185d107619414a8bcc15ad496547dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.420910 1203 tablet_bootstrap.cc:492] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:22.421046 1203 ts_tablet_manager.cc:1403] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.007s
I20251214 20:52:22.421737 1203 raft_consensus.cc:359] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:22.421900 1203 raft_consensus.cc:385] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.421949 1203 raft_consensus.cc:740] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.424969 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 119) took 1522 ms. Trace:
I20251214 20:52:22.425097 1977 rpcz_store.cc:276] 1214 20:52:20.902026 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.902066 (+ 40us) service_pool.cc:225] Handling call
1214 20:52:22.313747 (+1411681us) ts_tablet_manager.cc:1151] Deleting tablet 3c3962c33a294fa98f8bd485268aee71
1214 20:52:22.314126 (+ 379us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.331028 (+ 16902us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.337338 (+ 6310us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.424940 (+ 87602us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1411519}
I20251214 20:52:22.425273 1203 consensus_queue.cc:260] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:22.425453 1977 tablet_replica.cc:333] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:22.425652 1977 raft_consensus.cc:2243] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.425722 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 3c3962c33a294fa98f8bd485268aee71 (table default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]) successfully deleted
I20251214 20:52:22.425787 1203 ts_tablet_manager.cc:1434] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:22.425808 1977 raft_consensus.cc:2272] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.426342 1977 ts_tablet_manager.cc:1916] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.426888 1203 tablet_bootstrap.cc:492] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:22.432798 1203 tablet_bootstrap.cc:654] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.435585 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 172) took 1168 ms. Trace:
I20251214 20:52:22.435786 450 rpcz_store.cc:276] 1214 20:52:21.266717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.409865 (+1143148us) service_pool.cc:225] Handling call
1214 20:52:22.416312 (+ 6447us) spinlock_profiling.cc:243] Waited 6.14 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b1801c3 00007ff411c8a7c6 00007ff411c8a4f5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:22.417203 (+ 891us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:22.417728 (+ 525us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:22.435316 (+ 17588us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:22.435423 (+ 107us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:22.435524 (+ 101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.418414 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.418503 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.418507 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:22.418510 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.418590 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.418605 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.418610 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:22.418687 (+ 77us) tablet.cc:699] Row locks acquired
1214 20:52:22.418690 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.418725 (+ 35us) write_op.cc:270] Start()
1214 20:52:22.418757 (+ 32us) write_op.cc:276] Timestamp: P: 1765745542418643 usec, L: 0
1214 20:52:22.418761 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.418954 (+ 193us) log.cc:844] Serialized 433 byte log entry
1214 20:52:22.432329 (+ 13375us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.434757 (+ 2428us) write_op.cc:301] APPLY: starting
1214 20:52:22.434776 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.434818 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.434852 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.434970 (+ 118us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.435003 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:22.435011 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:22.435084 (+ 73us) log.cc:844] Serialized 29 byte log entry
1214 20:52:22.435166 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.435186 (+ 20us) write_op.cc:454] Released schema lock
1214 20:52:22.435192 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":14122752,"child_traces":[["op",{"apply.queue_time_us":2177,"apply.run_cpu_time_us":505,"apply.run_wall_time_us":693,"num_ops":2,"prepare.queue_time_us":146,"raft.queue_time_us":9975,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":685,"replication_time_us":13532}]]}
W20251214 20:52:22.437093 1769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 20:52:22.437283 1769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20251214 20:52:22.437285 450 master_service.cc:954] ReplaceTablet: received request to replace tablet 58ac06f852fc4abea5cb213926002ca1 from {username='slave'} at 127.0.0.1:50088
W20251214 20:52:22.437399 1769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 20:52:22.437701 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 76430us
I20251214 20:52:22.438436 1650 raft_consensus.cc:493] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.438475 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ac64f4fcb6a943668584658df19f950c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:22.438581 1650 raft_consensus.cc:515] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.439245 1650 leader_election.cc:290] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.439777 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 28d34670fdce4242a3feccf830a12880 (DEFAULT_TABLE table=default.Table_4d5d73b55b6245bf8594a4fc79f02adc [id=8316131458234291a566634324832ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.440176 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d34670fdce4242a3feccf830a12880. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.443359 1202 tablet_bootstrap.cc:492] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:22.443500 1202 ts_tablet_manager.cc:1403] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.132s user 0.012s sys 0.023s
I20251214 20:52:22.444598 746 tablet_service.cc:1505] Processing CreateTablet for tablet 28d34670fdce4242a3feccf830a12880 (DEFAULT_TABLE table=default.Table_4d5d73b55b6245bf8594a4fc79f02adc [id=8316131458234291a566634324832ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.444761 895 tablet_service.cc:1505] Processing CreateTablet for tablet 28d34670fdce4242a3feccf830a12880 (DEFAULT_TABLE table=default.Table_4d5d73b55b6245bf8594a4fc79f02adc [id=8316131458234291a566634324832ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.445075 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d34670fdce4242a3feccf830a12880. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.445204 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d34670fdce4242a3feccf830a12880. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.446910 1202 raft_consensus.cc:359] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.447085 1202 raft_consensus.cc:385] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.447180 1202 raft_consensus.cc:740] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.447423 1202 consensus_queue.cc:260] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:22.448009 851 leader_election.cc:343] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 762068d560ff47918dd9fd8791083433
I20251214 20:52:22.448123 1202 ts_tablet_manager.cc:1434] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:22.448382 1202 tablet_bootstrap.cc:492] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:22.451047 744 tablet_service.cc:1505] Processing CreateTablet for tablet 9bf9e65040af4fbeb701619fe27edfb6 (DEFAULT_TABLE table=default.Table_91b2da75e7104e0f95ed470c749f843f [id=70aba52f026e439f8bf0b12373148fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.451465 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf9e65040af4fbeb701619fe27edfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.453132 1202 tablet_bootstrap.cc:654] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.454080 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 9bf9e65040af4fbeb701619fe27edfb6 (DEFAULT_TABLE table=default.Table_91b2da75e7104e0f95ed470c749f843f [id=70aba52f026e439f8bf0b12373148fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.454523 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf9e65040af4fbeb701619fe27edfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.454772 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "762068d560ff47918dd9fd8791083433" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:22.455834 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 2d5920fdba794e5f864c595e7be278ab (DEFAULT_TABLE table=default.Table_873207e85b954e21b3d98ad4c02a0248 [id=451df68541d848278bfeda2dfce0ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.456182 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5920fdba794e5f864c595e7be278ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.456275 748 tablet_service.cc:1505] Processing CreateTablet for tablet 2d5920fdba794e5f864c595e7be278ab (DEFAULT_TABLE table=default.Table_873207e85b954e21b3d98ad4c02a0248 [id=451df68541d848278bfeda2dfce0ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.456295 751 tablet_service.cc:1505] Processing CreateTablet for tablet c9488c3505f5443998531cf749722be9 (DEFAULT_TABLE table=default.Table_d528fb3647e34ed5b6b9199d7c6db3a5 [id=f9962314a0e44479bcf6661cb8fcd650]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.456585 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5920fdba794e5f864c595e7be278ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.456764 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9488c3505f5443998531cf749722be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.457513 891 tablet_service.cc:1505] Processing CreateTablet for tablet 9bf9e65040af4fbeb701619fe27edfb6 (DEFAULT_TABLE table=default.Table_91b2da75e7104e0f95ed470c749f843f [id=70aba52f026e439f8bf0b12373148fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.457864 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf9e65040af4fbeb701619fe27edfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.460966 1977 ts_tablet_manager.cc:1929] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.461086 1977 log.cc:1199] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/2842f1dc0c374d24946f90dac1cf8aa5
I20251214 20:52:22.461403 1977 ts_tablet_manager.cc:1950] T 2842f1dc0c374d24946f90dac1cf8aa5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:22.462417 749 tablet_service.cc:1505] Processing CreateTablet for tablet f342dafae1d04be3ba38f7c0e4f5d058 (DEFAULT_TABLE table=default.Table_b849df93c2394733b20258632b62e6b4 [id=f33d449780b04f77b8cd399b5f5e91c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.462836 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f342dafae1d04be3ba38f7c0e4f5d058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.462731 1030 tablet_service.cc:1505] Processing CreateTablet for tablet c9488c3505f5443998531cf749722be9 (DEFAULT_TABLE table=default.Table_d528fb3647e34ed5b6b9199d7c6db3a5 [id=f9962314a0e44479bcf6661cb8fcd650]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.463248 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9488c3505f5443998531cf749722be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.456508 881 tablet_service.cc:1505] Processing CreateTablet for tablet 2d5920fdba794e5f864c595e7be278ab (DEFAULT_TABLE table=default.Table_873207e85b954e21b3d98ad4c02a0248 [id=451df68541d848278bfeda2dfce0ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.463770 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5920fdba794e5f864c595e7be278ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.464730 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2f9def8e4fd449599820a63845397750 (table default.Table_99e702b2f3fc436db82c793284a1670c [id=a9ff75b0f47f4601b2d489c3e3d6fce0]). Expected version 1 got 0
I20251214 20:52:22.465054 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:22.465216 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d01db63fd5d2474a9defdce0b98fc29d (table default.Table_13f016135de44cf78187fbc4f9f8c622 [id=fc160acc971e4f7d8290560ee8aab065]). Expected version 1 got 0
I20251214 20:52:22.465333 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:22.465492 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_76f578444d71410a93c3f78ff50b7951 [id=6157817e96d642d09f9941ad7f7e82ee]). Expected version 2 got 0
I20251214 20:52:22.465613 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:22.465822 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8e11723c042a4637839f1b13bed0935a (table default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]). Expected version 1 got 0
I20251214 20:52:22.466001 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:22.466334 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 566c6e4761d5486d964e276d151e32b0 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 2 got 0
I20251214 20:52:22.466995 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:22.467254 753 tablet_service.cc:1505] Processing CreateTablet for tablet 57a6232564194cbba12182f3a29c2ebb (DEFAULT_TABLE table=default.Table_9321256005b04155b2f5a451ce5065fe [id=c5a7a8665d6647bdb19ed6ef27907fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.467254 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 9da5e98c6f4843449886176e9fcfa8e1 (table default.Table_55f303d33ec84a269232fc1ba8f2d182 [id=5ded9fe825ae4308b5bd4b21c16634c8]). Expected version 1 got 0
I20251214 20:52:22.467440 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 57a6232564194cbba12182f3a29c2ebb (DEFAULT_TABLE table=default.Table_9321256005b04155b2f5a451ce5065fe [id=c5a7a8665d6647bdb19ed6ef27907fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.467576 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a6232564194cbba12182f3a29c2ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.467598 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d2688515bd1441fbad7516f8343ca775 (table default.Table_d2beb8dc1b054cada5e4c2ec9fabc8d4 [id=03997e2eab154ef18b2e3774b5984764]). Expected version 1 got 0
I20251214 20:52:22.467777 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a6232564194cbba12182f3a29c2ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.467972 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]). Expected version 1 got 0
I20251214 20:52:22.468202 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:22.468357 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:22.468874 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:22.469014 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
I20251214 20:52:22.469319 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:22.469583 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 1 got 0
I20251214 20:52:22.470022 894 tablet_service.cc:1505] Processing CreateTablet for tablet f342dafae1d04be3ba38f7c0e4f5d058 (DEFAULT_TABLE table=default.Table_b849df93c2394733b20258632b62e6b4 [id=f33d449780b04f77b8cd399b5f5e91c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.470407 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f342dafae1d04be3ba38f7c0e4f5d058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.470584 883 tablet_service.cc:1505] Processing CreateTablet for tablet 57a6232564194cbba12182f3a29c2ebb (DEFAULT_TABLE table=default.Table_9321256005b04155b2f5a451ce5065fe [id=c5a7a8665d6647bdb19ed6ef27907fee]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:22.470738 848 leader_election.cc:343] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.470947 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a6232564194cbba12182f3a29c2ebb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:22.471107 848 leader_election.cc:343] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.471222 848 leader_election.cc:304] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:22.471441 1650 raft_consensus.cc:2749] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:22.473217 740 tablet_service.cc:1505] Processing CreateTablet for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf (DEFAULT_TABLE table=default.Table_3f7396ac07d649fa8d342ae05765da80 [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.473570 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.476444 890 tablet_service.cc:1505] Processing CreateTablet for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf (DEFAULT_TABLE table=default.Table_3f7396ac07d649fa8d342ae05765da80 [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.476832 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.477082 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf (DEFAULT_TABLE table=default.Table_3f7396ac07d649fa8d342ae05765da80 [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.477430 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.477970 741 tablet_service.cc:1505] Processing CreateTablet for tablet cc8d725e6ac440899fc1cde2f7b546aa (DEFAULT_TABLE table=default.Table_9a53d238203b418eabb29093faef194c [id=978e62482f114f2b99e9707aaa9a5e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.478318 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc8d725e6ac440899fc1cde2f7b546aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.479009 894 tablet_service.cc:1505] Processing CreateTablet for tablet cc8d725e6ac440899fc1cde2f7b546aa (DEFAULT_TABLE table=default.Table_9a53d238203b418eabb29093faef194c [id=978e62482f114f2b99e9707aaa9a5e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.479374 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc8d725e6ac440899fc1cde2f7b546aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.479943 743 tablet_service.cc:1505] Processing CreateTablet for tablet 532c44785f204f988518c6fd628f76ca (DEFAULT_TABLE table=default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.480114 1020 tablet_service.cc:1505] Processing CreateTablet for tablet cc8d725e6ac440899fc1cde2f7b546aa (DEFAULT_TABLE table=default.Table_9a53d238203b418eabb29093faef194c [id=978e62482f114f2b99e9707aaa9a5e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.480341 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532c44785f204f988518c6fd628f76ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.480406 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc8d725e6ac440899fc1cde2f7b546aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.482344 757 tablet_service.cc:1505] Processing CreateTablet for tablet d745583012b4465090d85c75470e0bc2 (DEFAULT_TABLE table=default.Table_9e2910a477ea4326b888fa02a5fa5a79 [id=836ac984aa9549d88b3ecb1f69a3b8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.482708 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d745583012b4465090d85c75470e0bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.483541 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 532c44785f204f988518c6fd628f76ca (DEFAULT_TABLE table=default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.483860 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532c44785f204f988518c6fd628f76ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.484164 883 tablet_service.cc:1505] Processing CreateTablet for tablet d745583012b4465090d85c75470e0bc2 (DEFAULT_TABLE table=default.Table_9e2910a477ea4326b888fa02a5fa5a79 [id=836ac984aa9549d88b3ecb1f69a3b8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.484484 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d745583012b4465090d85c75470e0bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.485136 1034 tablet_service.cc:1505] Processing CreateTablet for tablet d745583012b4465090d85c75470e0bc2 (DEFAULT_TABLE table=default.Table_9e2910a477ea4326b888fa02a5fa5a79 [id=836ac984aa9549d88b3ecb1f69a3b8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.485442 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d745583012b4465090d85c75470e0bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.487895 1024 tablet_service.cc:1505] Processing CreateTablet for tablet f342dafae1d04be3ba38f7c0e4f5d058 (DEFAULT_TABLE table=default.Table_b849df93c2394733b20258632b62e6b4 [id=f33d449780b04f77b8cd399b5f5e91c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.488310 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f342dafae1d04be3ba38f7c0e4f5d058. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.488617 895 tablet_service.cc:1505] Processing CreateTablet for tablet c9488c3505f5443998531cf749722be9 (DEFAULT_TABLE table=default.Table_d528fb3647e34ed5b6b9199d7c6db3a5 [id=f9962314a0e44479bcf6661cb8fcd650]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.488992 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9488c3505f5443998531cf749722be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.492439 750 tablet_service.cc:1505] Processing CreateTablet for tablet 43fe3bb1ed3941ce9078376f8f478408 (DEFAULT_TABLE table=default.Table_eb4f37e15d9a4b1b9e3892c7891329bc [id=e416e18f223e401f8652da2b9f222cdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.492828 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43fe3bb1ed3941ce9078376f8f478408. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.492972 891 tablet_service.cc:1505] Processing CreateTablet for tablet 532c44785f204f988518c6fd628f76ca (DEFAULT_TABLE table=default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.493340 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 532c44785f204f988518c6fd628f76ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.494300 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (table default.Table_89c0478b97144f35b1a0adeb64be2d9f [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]). Expected version 1 got 0
I20251214 20:52:22.494580 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:22.494801 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:22.495221 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:22.495517 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:22.495668 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:22.496074 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:22.496327 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:22.496587 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_41d7d906f2714c57832f29b1a2692574 [id=9553b46856c94003a293135c814a7df7]). Expected version 1 got 0
I20251214 20:52:22.496865 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet feb19ea389af4ea6bb158a0e8aaeb094 (table default.Table_7660878c81e04c17b9b6fa8d61ef43a6 [id=59d9716c5cee42c391018111323c5181]). Expected version 1 got 0
I20251214 20:52:22.497294 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d171301941fa43908e05984c3b016007 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:22.497486 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 727e7b61ae3942098712720d344c5310 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]). Expected version 1 got 0
I20251214 20:52:22.500895 738 tablet_service.cc:1505] Processing CreateTablet for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738 (DEFAULT_TABLE table=default.Table_d9749ec3d4b8476a9e3d18ff2d300544 [id=a56279b96c2d45af90277c7636192936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.501173 742 tablet_service.cc:1505] Processing CreateTablet for tablet 34df3a34b15b4fe19bd12f785b07a033 (DEFAULT_TABLE table=default.Table_1e4dafc4ff8940ea90956623c915e3fd [id=be1be47d388347e3a6b7a80105688be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.501412 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.501765 890 tablet_service.cc:1505] Processing CreateTablet for tablet 34df3a34b15b4fe19bd12f785b07a033 (DEFAULT_TABLE table=default.Table_1e4dafc4ff8940ea90956623c915e3fd [id=be1be47d388347e3a6b7a80105688be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.502218 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34df3a34b15b4fe19bd12f785b07a033. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.502472 894 tablet_service.cc:1505] Processing CreateTablet for tablet 43fe3bb1ed3941ce9078376f8f478408 (DEFAULT_TABLE table=default.Table_eb4f37e15d9a4b1b9e3892c7891329bc [id=e416e18f223e401f8652da2b9f222cdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.502743 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 120) took 1600 ms. Trace:
I20251214 20:52:22.502878 1977 rpcz_store.cc:276] 1214 20:52:20.902369 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.902438 (+ 69us) service_pool.cc:225] Handling call
1214 20:52:22.425398 (+1522960us) ts_tablet_manager.cc:1151] Deleting tablet 2842f1dc0c374d24946f90dac1cf8aa5
1214 20:52:22.425408 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.457808 (+ 32400us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.460867 (+ 3059us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.502710 (+ 41843us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1522810}
I20251214 20:52:22.503334 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 2842f1dc0c374d24946f90dac1cf8aa5 (table default.Table_a2a64eab2d9a493da6fee1051eadd37d [id=d929401b9c4c4b4ba21353fb8a8ecc3e]) successfully deleted
I20251214 20:52:22.503362 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 121) took 1600 ms. Trace:
I20251214 20:52:22.503434 1977 rpcz_store.cc:276] 1214 20:52:20.902714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.902746 (+ 32us) service_pool.cc:225] Handling call
1214 20:52:22.503244 (+1600498us) ts_tablet_manager.cc:1151] Deleting tablet c2ae9dc8ead8403cb7215af8778872f0
1214 20:52:22.503255 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.503346 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600350}
W20251214 20:52:22.503829 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c2ae9dc8ead8403cb7215af8778872f0 because the tablet was not found. No further retry: Not found: Tablet not found: c2ae9dc8ead8403cb7215af8778872f0
W20251214 20:52:22.504056 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 0c973e0111804b8b869068bdcc5869bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c973e0111804b8b869068bdcc5869bf already in progress: creating tablet
I20251214 20:52:22.503808 881 tablet_service.cc:1505] Processing CreateTablet for tablet 812642cfcff940f899ef9d3644909ec4 (DEFAULT_TABLE table=default.Table_08a9cb7994804ae586d3a7e3de4adb7d [id=74f23f7675974c82953d757dd563179a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.502801 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43fe3bb1ed3941ce9078376f8f478408. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.503908 1977 tablet_replica.cc:333] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:22.507232 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812642cfcff940f899ef9d3644909ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.507390 1977 raft_consensus.cc:2243] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.507588 1977 raft_consensus.cc:2272] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.508242 1977 ts_tablet_manager.cc:1916] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.508296 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 812642cfcff940f899ef9d3644909ec4 (DEFAULT_TABLE table=default.Table_08a9cb7994804ae586d3a7e3de4adb7d [id=74f23f7675974c82953d757dd563179a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.508709 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812642cfcff940f899ef9d3644909ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.509542 880 tablet_service.cc:1505] Processing CreateTablet for tablet 147c985fa7c94f7a9af57917bf17b29f (DEFAULT_TABLE table=default.Table_14f3cdeac59a4c20988d0e23744d2fbd [id=d53e0d2c6a7b4dc89f93ddd315732229]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.509886 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147c985fa7c94f7a9af57917bf17b29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.501765 883 tablet_service.cc:1505] Processing CreateTablet for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738 (DEFAULT_TABLE table=default.Table_d9749ec3d4b8476a9e3d18ff2d300544 [id=a56279b96c2d45af90277c7636192936]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.517884 885 tablet_service.cc:1505] Processing CreateTablet for tablet bd1161eebe884fe6b347bf517269c047 (DEFAULT_TABLE table=default.Table_de80fc127aa549ca931866a310858ddb [id=4bc7334a27824731b0d62be48aa40b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.517894 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 147c985fa7c94f7a9af57917bf17b29f (DEFAULT_TABLE table=default.Table_14f3cdeac59a4c20988d0e23744d2fbd [id=d53e0d2c6a7b4dc89f93ddd315732229]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.518396 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147c985fa7c94f7a9af57917bf17b29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.524238 1202 tablet_bootstrap.cc:492] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:22.524426 1202 ts_tablet_manager.cc:1403] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.007s
I20251214 20:52:22.525245 1202 raft_consensus.cc:359] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.525439 1202 raft_consensus.cc:385] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.525491 1202 raft_consensus.cc:740] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.530045 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 58ac06f852fc4abea5cb213926002ca1
I20251214 20:52:22.530575 450 catalog_manager.cc:6724] ReplaceTablet: tablet 58ac06f852fc4abea5cb213926002ca1 deleted and replaced by tablet 248e3666fd13434e97378eb5a98dfadd
I20251214 20:52:22.530684 450 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 173) took 1171 ms. Trace:
I20251214 20:52:22.530790 450 rpcz_store.cc:276] 1214 20:52:21.359033 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.437198 (+1078165us) service_pool.cc:225] Handling call
1214 20:52:22.530655 (+ 93457us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.525073 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.525212 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.525218 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:22.525221 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.525347 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.525361 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.525366 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:22.525706 (+ 340us) tablet.cc:699] Row locks acquired
1214 20:52:22.525709 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.525753 (+ 44us) write_op.cc:270] Start()
1214 20:52:22.525791 (+ 38us) write_op.cc:276] Timestamp: P: 1765745542525648 usec, L: 0
1214 20:52:22.525794 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.526165 (+ 371us) log.cc:844] Serialized 622 byte log entry
1214 20:52:22.529161 (+ 2996us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.529318 (+ 157us) write_op.cc:301] APPLY: starting
1214 20:52:22.529332 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.529376 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.529399 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.529501 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.529561 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.529568 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.529637 (+ 69us) log.cc:844] Serialized 29 byte log entry
1214 20:52:22.529701 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.529722 (+ 21us) write_op.cc:454] Released schema lock
1214 20:52:22.529729 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1361664,"child_traces":[["op",{"apply.queue_time_us":112,"apply.run_cpu_time_us":470,"apply.run_wall_time_us":470,"num_ops":2,"prepare.queue_time_us":82,"raft.queue_time_us":38,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":237,"replication_time_us":3339}]]}
I20251214 20:52:22.533046 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd1161eebe884fe6b347bf517269c047. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.532831 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 43fe3bb1ed3941ce9078376f8f478408 (DEFAULT_TABLE table=default.Table_eb4f37e15d9a4b1b9e3892c7891329bc [id=e416e18f223e401f8652da2b9f222cdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.533569 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43fe3bb1ed3941ce9078376f8f478408. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.533809 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.534214 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_02bb72c146234757b420555977074934"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.534857 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02bb72c146234757b420555977074934 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.536964 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738 (DEFAULT_TABLE table=default.Table_d9749ec3d4b8476a9e3d18ff2d300544 [id=a56279b96c2d45af90277c7636192936]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:22.537202 991 consensus_peers.cc:597] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb698ff30fcc47d09000cc68d2c5225b. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:22.537528 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.539480 1202 consensus_queue.cc:260] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.542711 1977 ts_tablet_manager.cc:1929] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:22.542833 1977 log.cc:1199] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/e16fe751f10f4616a00d0af811998729
I20251214 20:52:22.544230 1202 ts_tablet_manager.cc:1434] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20251214 20:52:22.544212 452 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 6eca34627bbc4a3f9915be72f3128370
I20251214 20:52:22.544734 452 catalog_manager.cc:6724] ReplaceTablet: tablet 6eca34627bbc4a3f9915be72f3128370 deleted and replaced by tablet 6d535d8f71b44c81bbba6937e804d900
I20251214 20:52:22.544865 452 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 169) took 1317 ms. Trace:
I20251214 20:52:22.544992 1202 tablet_bootstrap.cc:492] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:22.544967 452 rpcz_store.cc:276] 1214 20:52:21.227336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.320408 (+1093072us) service_pool.cc:225] Handling call
1214 20:52:22.544834 (+224426us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.537746 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.537863 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.537868 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:22.537871 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.537999 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.538014 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.538019 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:22.538118 (+ 99us) tablet.cc:699] Row locks acquired
1214 20:52:22.538121 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.538161 (+ 40us) write_op.cc:270] Start()
1214 20:52:22.538195 (+ 34us) write_op.cc:276] Timestamp: P: 1765745542538056 usec, L: 0
1214 20:52:22.538200 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.538398 (+ 198us) log.cc:844] Serialized 622 byte log entry
1214 20:52:22.543173 (+ 4775us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.543447 (+ 274us) write_op.cc:301] APPLY: starting
1214 20:52:22.543461 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.543499 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.543532 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.543621 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.543691 (+ 70us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.543698 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.543765 (+ 67us) log.cc:844] Serialized 29 byte log entry
1214 20:52:22.543818 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.543834 (+ 16us) write_op.cc:454] Released schema lock
1214 20:52:22.543840 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.543917 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.543920 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":193,"apply.run_cpu_time_us":450,"apply.run_wall_time_us":450,"num_ops":2,"prepare.queue_time_us":11855,"prepare.run_cpu_time_us":818,"prepare.run_wall_time_us":3018,"raft.queue_time_us":291,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":251,"replication_time_us":4909,"spinlock_wait_cycles":1920}]]}
I20251214 20:52:22.545692 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_dce3448de74f4b92b7cc3de061c34dc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.546276 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dce3448de74f4b92b7cc3de061c34dc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.551023 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ac06f852fc4abea5cb213926002ca1 with delete_type TABLET_DATA_DELETED (replaced by tablet 248e3666fd13434e97378eb5a98dfadd) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.543579 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 34df3a34b15b4fe19bd12f785b07a033 (DEFAULT_TABLE table=default.Table_1e4dafc4ff8940ea90956623c915e3fd [id=be1be47d388347e3a6b7a80105688be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.555801 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34df3a34b15b4fe19bd12f785b07a033. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.557011 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 174) took 1155 ms. Trace:
I20251214 20:52:22.557607 450 rpcz_store.cc:276] 1214 20:52:21.401287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.531887 (+1130600us) service_pool.cc:225] Handling call
1214 20:52:22.534164 (+ 2277us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b1801c3 00007ff411c8a7c6 00007ff411c8a4f5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:22.534942 (+ 778us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:22.535442 (+ 500us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:22.556769 (+ 21327us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:22.556871 (+ 102us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:22.556952 (+ 81us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.540797 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.540900 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.540905 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:22.540908 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.540992 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.541002 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.541006 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:22.541074 (+ 68us) tablet.cc:699] Row locks acquired
1214 20:52:22.541076 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:22.541106 (+ 30us) write_op.cc:270] Start()
1214 20:52:22.541141 (+ 35us) write_op.cc:276] Timestamp: P: 1765745542541003 usec, L: 0
1214 20:52:22.541144 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.541318 (+ 174us) log.cc:844] Serialized 433 byte log entry
1214 20:52:22.555384 (+ 14066us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.555517 (+ 133us) write_op.cc:301] APPLY: starting
1214 20:52:22.555533 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.555577 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.555608 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.555714 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.555749 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:22.555756 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.555837 (+ 81us) log.cc:844] Serialized 29 byte log entry
1214 20:52:22.555896 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.555917 (+ 21us) write_op.cc:454] Released schema lock
1214 20:52:22.555924 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.556000 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.556003 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":4613376,"child_traces":[["op",{"apply.queue_time_us":84,"apply.run_cpu_time_us":474,"apply.run_wall_time_us":474,"num_ops":2,"prepare.queue_time_us":2103,"prepare.run_cpu_time_us":620,"prepare.run_wall_time_us":895,"raft.queue_time_us":212,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":14207,"spinlock_wait_cycles":64384}]]}
I20251214 20:52:22.559392 1202 tablet_bootstrap.cc:654] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.559459 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ac06f852fc4abea5cb213926002ca1 with delete_type TABLET_DATA_DELETED (replaced by tablet 248e3666fd13434e97378eb5a98dfadd) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:22.560014 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_edb8b6390dba4303b887b41508f5f525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.560684 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edb8b6390dba4303b887b41508f5f525 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.561169 1977 ts_tablet_manager.cc:1950] T e16fe751f10f4616a00d0af811998729 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:22.561249 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 6eca34627bbc4a3f9915be72f3128370 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d535d8f71b44c81bbba6937e804d900) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.562872 1977 tablet_replica.cc:333] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:22.562875 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet e16fe751f10f4616a00d0af811998729 (table default.Table_7923a9cba4ce4b27a12617e394ffbf2e [id=8fd9be1206a74aa1921e0dade8be3774]) successfully deleted
I20251214 20:52:22.563104 1977 raft_consensus.cc:2243] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:22.563284 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 17550eb0ee6b476f8a8c304451f678b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17550eb0ee6b476f8a8c304451f678b2 already in progress: creating tablet
I20251214 20:52:22.563299 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 6eca34627bbc4a3f9915be72f3128370 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d535d8f71b44c81bbba6937e804d900) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:22.563562 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2d164747cb294bd4b126b4420231867d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d164747cb294bd4b126b4420231867d already in progress: creating tablet
W20251214 20:52:22.563788 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8eb1006b0ba440dcba2f9de04dca2702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8eb1006b0ba440dcba2f9de04dca2702 already in progress: creating tablet
I20251214 20:52:22.559590 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
W20251214 20:52:22.564013 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet dfe50a6219134431b0a025317932609b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfe50a6219134431b0a025317932609b already in progress: creating tablet
W20251214 20:52:22.564536 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet fe24e4f4e74040249068505058c15622 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24e4f4e74040249068505058c15622 already in progress: creating tablet
I20251214 20:52:22.564343 1019 tablet_service.cc:1505] Processing CreateTablet for tablet bd1161eebe884fe6b347bf517269c047 (DEFAULT_TABLE table=default.Table_de80fc127aa549ca931866a310858ddb [id=4bc7334a27824731b0d62be48aa40b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.564636 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
I20251214 20:52:22.564889 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 9da5e98c6f4843449886176e9fcfa8e1 (table default.Table_55f303d33ec84a269232fc1ba8f2d182 [id=5ded9fe825ae4308b5bd4b21c16634c8]). Expected version 1 got 0
I20251214 20:52:22.565164 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 2f9def8e4fd449599820a63845397750 (table default.Table_99e702b2f3fc436db82c793284a1670c [id=a9ff75b0f47f4601b2d489c3e3d6fce0]). Expected version 1 got 0
I20251214 20:52:22.565245 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd1161eebe884fe6b347bf517269c047. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.565413 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:22.565734 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 727e7b61ae3942098712720d344c5310 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]). Expected version 1 got 0
I20251214 20:52:22.566562 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:22.566730 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_1f9ca4a533f64c5ebb5b98f0e25dfe76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:22.567164 1977 raft_consensus.cc:2272] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.567225 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
W20251214 20:52:22.567373 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f9ca4a533f64c5ebb5b98f0e25dfe76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.567405 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d171301941fa43908e05984c3b016007 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:22.567726 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:22.567790 1977 ts_tablet_manager.cc:1916] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.567924 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d01db63fd5d2474a9defdce0b98fc29d (table default.Table_13f016135de44cf78187fbc4f9f8c622 [id=fc160acc971e4f7d8290560ee8aab065]). Expected version 1 got 0
I20251214 20:52:22.568334 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 1 got 0
I20251214 20:52:22.568584 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
I20251214 20:52:22.568935 448 catalog_manager.cc:5654] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:22.569314 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:22.569599 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:22.569980 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet feb19ea389af4ea6bb158a0e8aaeb094 (table default.Table_7660878c81e04c17b9b6fa8d61ef43a6 [id=59d9716c5cee42c391018111323c5181]). Expected version 1 got 0
I20251214 20:52:22.570657 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:22.570884 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (table default.Table_89c0478b97144f35b1a0adeb64be2d9f [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]). Expected version 1 got 0
I20251214 20:52:22.571393 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]). Expected version 1 got 0
I20251214 20:52:22.571820 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 566c6e4761d5486d964e276d151e32b0 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 2 got 0
I20251214 20:52:22.572023 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_41d7d906f2714c57832f29b1a2692574 [id=9553b46856c94003a293135c814a7df7]). Expected version 1 got 0
I20251214 20:52:22.572520 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:22.572914 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d2688515bd1441fbad7516f8343ca775 (table default.Table_d2beb8dc1b054cada5e4c2ec9fabc8d4 [id=03997e2eab154ef18b2e3774b5984764]). Expected version 1 got 0
I20251214 20:52:22.573187 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 8e11723c042a4637839f1b13bed0935a (table default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]). Expected version 1 got 0
I20251214 20:52:22.573308 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_e82b745b92174d32a0d7bc8f779a6558" } modify_external_catalogs: true
I20251214 20:52:22.573437 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:22.573513 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_e82b745b92174d32a0d7bc8f779a6558" } modify_external_catalogs: true
I20251214 20:52:22.573673 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:22.573956 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:22.574158 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:22.574558 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:22.574806 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
W20251214 20:52:22.581554 1769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 20:52:22.581825 1977 ts_tablet_manager.cc:1929] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:22.581979 1977 log.cc:1199] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/02e256d4537249a394b80dc36beefef4
I20251214 20:52:22.582302 1977 ts_tablet_manager.cc:1950] T 02e256d4537249a394b80dc36beefef4 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:22.587487 1650 raft_consensus.cc:493] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.587669 1650 raft_consensus.cc:515] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.588315 1650 leader_election.cc:290] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.589108 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca57963bcfb54eec9fcbed82757d8387" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:22.590615 848 leader_election.cc:343] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.591687 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 02e256d4537249a394b80dc36beefef4 (table default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]) successfully deleted
I20251214 20:52:22.591764 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca57963bcfb54eec9fcbed82757d8387" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:22.592010 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 58ac06f852fc4abea5cb213926002ca1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58ac06f852fc4abea5cb213926002ca1 already in progress: creating tablet
W20251214 20:52:22.592272 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 6eca34627bbc4a3f9915be72f3128370 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6eca34627bbc4a3f9915be72f3128370 already in progress: creating tablet
I20251214 20:52:22.592854 1856 ts_tablet_manager.cc:1929] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:22.593122 1856 log.cc:1199] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/604b7c6fe3b24daca70711a1990af556
W20251214 20:52:22.593667 851 leader_election.cc:343] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.593776 851 leader_election.cc:304] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:22.594359 452 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6a61fe3bf1734a888048db0d20abac60" } modify_external_catalogs: true
I20251214 20:52:22.594579 452 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6a61fe3bf1734a888048db0d20abac60" } modify_external_catalogs: true
I20251214 20:52:22.594780 1650 raft_consensus.cc:2749] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:22.603562 1203 tablet_bootstrap.cc:492] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:22.603833 1203 ts_tablet_manager.cc:1403] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.177s user 0.015s sys 0.010s
I20251214 20:52:22.604842 1203 raft_consensus.cc:359] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.605257 1203 raft_consensus.cc:385] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.605352 1203 raft_consensus.cc:740] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.605657 1203 consensus_queue.cc:260] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.608085 1203 ts_tablet_manager.cc:1434] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:22.608520 1203 tablet_bootstrap.cc:492] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:22.608534 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_41d7d906f2714c57832f29b1a2692574 [id=9553b46856c94003a293135c814a7df7]). Expected version 1 got 0
I20251214 20:52:22.609731 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet feb19ea389af4ea6bb158a0e8aaeb094 (table default.Table_7660878c81e04c17b9b6fa8d61ef43a6 [id=59d9716c5cee42c391018111323c5181]). Expected version 1 got 0
I20251214 20:52:22.610525 1203 tablet_bootstrap.cc:654] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.611382 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:22.611827 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:22.612051 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:22.612282 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 727e7b61ae3942098712720d344c5310 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]). Expected version 1 got 0
I20251214 20:52:22.612546 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:22.613095 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:22.613505 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]). Expected version 1 got 0
I20251214 20:52:22.613744 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (table default.Table_89c0478b97144f35b1a0adeb64be2d9f [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]). Expected version 1 got 0
I20251214 20:52:22.614087 447 catalog_manager.cc:5654] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:22.615114 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:22.615733 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
I20251214 20:52:22.616031 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:22.616390 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]). Expected version 1 got 0
I20251214 20:52:22.616669 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:22.616894 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:22.617111 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d2688515bd1441fbad7516f8343ca775 (table default.Table_d2beb8dc1b054cada5e4c2ec9fabc8d4 [id=03997e2eab154ef18b2e3774b5984764]). Expected version 1 got 0
I20251214 20:52:22.618429 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:22.618783 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:22.619347 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 566c6e4761d5486d964e276d151e32b0 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 2 got 0
I20251214 20:52:22.620133 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:22.620363 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d171301941fa43908e05984c3b016007 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:22.620592 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8e11723c042a4637839f1b13bed0935a (table default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]). Expected version 1 got 0
I20251214 20:52:22.621100 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:22.621548 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d01db63fd5d2474a9defdce0b98fc29d (table default.Table_13f016135de44cf78187fbc4f9f8c622 [id=fc160acc971e4f7d8290560ee8aab065]). Expected version 1 got 0
I20251214 20:52:22.621856 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 9da5e98c6f4843449886176e9fcfa8e1 (table default.Table_55f303d33ec84a269232fc1ba8f2d182 [id=5ded9fe825ae4308b5bd4b21c16634c8]). Expected version 1 got 0
I20251214 20:52:22.622243 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2f9def8e4fd449599820a63845397750 (table default.Table_99e702b2f3fc436db82c793284a1670c [id=a9ff75b0f47f4601b2d489c3e3d6fce0]). Expected version 1 got 0
I20251214 20:52:22.622465 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:22.622813 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 1 got 0
I20251214 20:52:22.623251 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]). Expected version 2 got 0
W20251214 20:52:22.623742 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80098b7c24824d129b8f25daf2115460 consensus state has no leader
I20251214 20:52:22.623919 451 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 170) took 1394 ms. Trace:
I20251214 20:52:22.624040 451 rpcz_store.cc:276] 1214 20:52:21.229527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.355636 (+1126109us) service_pool.cc:225] Handling call
1214 20:52:22.356580 (+ 944us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:22.356828 (+ 248us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:22.357034 (+ 206us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:22.357123 (+ 89us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:22.380847 (+ 23724us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:22.623883 (+243036us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.359364 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.359474 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.359479 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:22.359482 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.359611 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.359622 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.359627 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.359690 (+ 63us) tablet.cc:699] Row locks acquired
1214 20:52:22.359693 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.359726 (+ 33us) write_op.cc:270] Start()
1214 20:52:22.359755 (+ 29us) write_op.cc:276] Timestamp: P: 1765745542359659 usec, L: 0
1214 20:52:22.359758 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.359920 (+ 162us) log.cc:844] Serialized 405 byte log entry
1214 20:52:22.379385 (+ 19465us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.380394 (+ 1009us) write_op.cc:301] APPLY: starting
1214 20:52:22.380407 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.380440 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.380465 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.380509 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.380539 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.380545 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:22.380602 (+ 57us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.380660 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.380674 (+ 14us) write_op.cc:454] Released schema lock
1214 20:52:22.380679 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.380745 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.380749 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":957,"apply.run_cpu_time_us":345,"apply.run_wall_time_us":345,"num_ops":1,"prepare.queue_time_us":1781,"prepare.run_cpu_time_us":689,"prepare.run_wall_time_us":20253,"raft.queue_time_us":646,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":19594}]]}
I20251214 20:52:22.624907 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e" } modify_external_catalogs: true
I20251214 20:52:22.625123 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e" } modify_external_catalogs: true
I20251214 20:52:22.626940 446 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_3c676f861c28408f99faac2ab8f7eec1 [id=ef22180b36a241fea13d0e622c02b50a] alter complete (version 3)
W20251214 20:52:22.627795 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a23561b9f4e48bbbd796e14fafec086 consensus state has no leader
I20251214 20:52:22.627975 453 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 166) took 1440 ms. Trace:
I20251214 20:52:22.628083 453 rpcz_store.cc:276] 1214 20:52:21.187335 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.243603 (+1056268us) service_pool.cc:225] Handling call
1214 20:52:22.279934 (+ 36331us) spinlock_profiling.cc:243] Waited 36.1 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b182456 00007ff411c8e146 00007ff411c8de75 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:22.280307 (+ 373us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:22.280534 (+ 227us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:22.285860 (+ 5326us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:22.285992 (+ 132us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:22.316201 (+ 30209us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:22.627934 (+311733us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.308835 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.308954 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.308961 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:22.308964 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.309052 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.309063 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.309068 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.309125 (+ 57us) tablet.cc:699] Row locks acquired
1214 20:52:22.309128 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.309159 (+ 31us) write_op.cc:270] Start()
1214 20:52:22.309194 (+ 35us) write_op.cc:276] Timestamp: P: 1765745542309102 usec, L: 0
1214 20:52:22.309199 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.309376 (+ 177us) log.cc:844] Serialized 405 byte log entry
1214 20:52:22.315295 (+ 5919us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.315760 (+ 465us) write_op.cc:301] APPLY: starting
1214 20:52:22.315771 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.315798 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.315826 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.315856 (+ 30us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.315877 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.315883 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:22.315945 (+ 62us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.315988 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.316003 (+ 15us) write_op.cc:454] Released schema lock
1214 20:52:22.316010 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.316084 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.316088 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":82949888,"child_traces":[["op",{"apply.queue_time_us":409,"apply.run_cpu_time_us":318,"apply.run_wall_time_us":317,"num_ops":1,"prepare.queue_time_us":22175,"prepare.run_cpu_time_us":667,"prepare.run_wall_time_us":787,"raft.queue_time_us":36,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":6067}]]}
I20251214 20:52:22.629004 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_59405cb9326245918d9cf6fb65ddbf40" } modify_external_catalogs: true
I20251214 20:52:22.629227 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_59405cb9326245918d9cf6fb65ddbf40" } modify_external_catalogs: true
I20251214 20:52:22.626711 449 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9] alter complete (version 1)
I20251214 20:52:22.637346 1650 raft_consensus.cc:493] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.637542 1650 raft_consensus.cc:515] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.638289 1650 leader_election.cc:290] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.639204 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2accf8a160584c82bb695a52e3f2b05c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:22.639858 848 leader_election.cc:343] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.640168 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2accf8a160584c82bb695a52e3f2b05c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:22.640694 851 leader_election.cc:343] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.640784 851 leader_election.cc:304] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:22.641193 1650 raft_consensus.cc:2749] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:22.659770 1963 raft_consensus.cc:493] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2e8fd1cb731446d88f8e510c7a52999)
I20251214 20:52:22.659770 740 tablet_service.cc:1505] Processing CreateTablet for tablet 812642cfcff940f899ef9d3644909ec4 (DEFAULT_TABLE table=default.Table_08a9cb7994804ae586d3a7e3de4adb7d [id=74f23f7675974c82953d757dd563179a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.660029 1963 raft_consensus.cc:515] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:22.660429 741 tablet_service.cc:1505] Processing CreateTablet for tablet 147c985fa7c94f7a9af57917bf17b29f (DEFAULT_TABLE table=default.Table_14f3cdeac59a4c20988d0e23744d2fbd [id=d53e0d2c6a7b4dc89f93ddd315732229]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.660808 1963 leader_election.cc:290] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:22.661154 773 raft_consensus.cc:3060] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.661255 774 raft_consensus.cc:3060] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.664302 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 6eca34627bbc4a3f9915be72f3128370 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d535d8f71b44c81bbba6937e804d900) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.664670 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34df3a34b15b4fe19bd12f785b07a033. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.664930 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 147c985fa7c94f7a9af57917bf17b29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.664865 755 tablet_service.cc:1505] Processing CreateTablet for tablet bd1161eebe884fe6b347bf517269c047 (DEFAULT_TABLE table=default.Table_de80fc127aa549ca931866a310858ddb [id=4bc7334a27824731b0d62be48aa40b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.665187 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd1161eebe884fe6b347bf517269c047. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.666132 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812642cfcff940f899ef9d3644909ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.667311 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ac06f852fc4abea5cb213926002ca1 with delete_type TABLET_DATA_DELETED (replaced by tablet 248e3666fd13434e97378eb5a98dfadd) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:22.670850 712 leader_election.cc:343] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: c16895ca08d94b56b5c4bc9352e330ef
I20251214 20:52:22.678689 2079 consensus_queue.cc:1048] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:22.682320 2079 consensus_queue.cc:1048] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:22.687402 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 274185a4115b458e8e9a376bdbcc870b
I20251214 20:52:22.688237 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b849df93c2394733b20258632b62e6b4" } modify_external_catalogs: true
I20251214 20:52:22.688338 1650 raft_consensus.cc:493] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.688427 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b849df93c2394733b20258632b62e6b4" } modify_external_catalogs: true
I20251214 20:52:22.688536 1650 raft_consensus.cc:515] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.689239 1650 leader_election.cc:290] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.689813 452 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 3b47ca1a03944cf1847d3578856dafa7
I20251214 20:52:22.690543 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_65d3039202a645e899e5f752e1e3db42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.691218 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65d3039202a645e899e5f752e1e3db42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.692466 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 5f4760585baf4b4ab3b9b94b5a748a61
I20251214 20:52:22.693208 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_dd6d020171b24861a33a63f13b128050"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.693678 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd6d020171b24861a33a63f13b128050 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.695649 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 274185a4115b458e8e9a376bdbcc870b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.695952 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b47ca1a03944cf1847d3578856dafa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.696105 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f4760585baf4b4ab3b9b94b5a748a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.696429 1977 tablet_replica.cc:333] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:22.696638 1977 raft_consensus.cc:2243] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:22.696779 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 274185a4115b458e8e9a376bdbcc870b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 274185a4115b458e8e9a376bdbcc870b already in progress: creating tablet
I20251214 20:52:22.696830 1977 raft_consensus.cc:2272] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:22.697108 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3b47ca1a03944cf1847d3578856dafa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b47ca1a03944cf1847d3578856dafa7 already in progress: creating tablet
I20251214 20:52:22.697141 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5da9deeedce0421c9aeaebd71ba11b1b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:22.697337 1056 raft_consensus.cc:2468] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:22.697841 851 leader_election.cc:304] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.698220 1650 raft_consensus.cc:2804] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:22.698300 1650 raft_consensus.cc:493] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:22.698361 1650 raft_consensus.cc:3060] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.698834 1977 ts_tablet_manager.cc:1916] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.701104 1650 raft_consensus.cc:515] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.705265 1977 ts_tablet_manager.cc:1929] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.705363 1977 log.cc:1199] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/5f4760585baf4b4ab3b9b94b5a748a61
I20251214 20:52:22.705325 1650 leader_election.cc:290] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.705907 1977 ts_tablet_manager.cc:1950] T 5f4760585baf4b4ab3b9b94b5a748a61 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:22.706480 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5da9deeedce0421c9aeaebd71ba11b1b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:22.706465 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5da9deeedce0421c9aeaebd71ba11b1b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:22.706673 1056 raft_consensus.cc:3060] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:22.707010 991 consensus_peers.cc:597] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:22.707111 848 leader_election.cc:343] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.707674 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5da9deeedce0421c9aeaebd71ba11b1b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:22.708163 848 leader_election.cc:343] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.708530 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]) successfully deleted
I20251214 20:52:22.709502 1056 raft_consensus.cc:2468] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:22.710007 851 leader_election.cc:304] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:22.710431 1650 raft_consensus.cc:2804] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:22.711323 1650 raft_consensus.cc:697] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:22.711658 1650 consensus_queue.cc:237] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.742837 1982 ts_tablet_manager.cc:1929] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:22.743029 1982 log.cc:1199] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/a10663bd306d40e7a6375d06d69be22d
I20251214 20:52:22.756255 1856 ts_tablet_manager.cc:1950] T 604b7c6fe3b24daca70711a1990af556 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:22.756417 1982 ts_tablet_manager.cc:1950] T a10663bd306d40e7a6375d06d69be22d P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:22.756790 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 110) took 2531 ms. Trace:
I20251214 20:52:22.756949 1982 rpcz_store.cc:276] 1214 20:52:20.225590 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.225652 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:22.326757 (+2101105us) ts_tablet_manager.cc:1151] Deleting tablet a10663bd306d40e7a6375d06d69be22d
1214 20:52:22.326772 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.406580 (+ 79808us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.742672 (+336092us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.756770 (+ 14098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1953183}
I20251214 20:52:22.757557 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 111) took 2531 ms. Trace:
I20251214 20:52:22.757799 1982 rpcz_store.cc:276] 1214 20:52:20.225972 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.226026 (+ 54us) service_pool.cc:225] Handling call
1214 20:52:22.757412 (+2531386us) ts_tablet_manager.cc:1151] Deleting tablet 9b1a937150ce4ecfa210c8465d682496
1214 20:52:22.757422 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.757540 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2383789}
I20251214 20:52:22.761809 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 139) took 1884 ms. Trace:
I20251214 20:52:22.762025 1856 rpcz_store.cc:276] 1214 20:52:20.877462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.877773 (+ 311us) service_pool.cc:225] Handling call
1214 20:52:22.323015 (+1445242us) ts_tablet_manager.cc:1151] Deleting tablet 604b7c6fe3b24daca70711a1990af556
1214 20:52:22.323031 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.410080 (+ 87049us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.592589 (+182509us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.761777 (+169188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1444891}
I20251214 20:52:22.763085 1856 tablet_replica.cc:333] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:22.763386 1856 raft_consensus.cc:2243] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.763473 1982 tablet_replica.cc:333] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:22.763614 1856 raft_consensus.cc:2272] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.764317 1856 ts_tablet_manager.cc:1916] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.755858 1206 raft_consensus.cc:385] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.765053 1206 raft_consensus.cc:740] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.766494 1206 consensus_queue.cc:260] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.767314 1206 ts_tablet_manager.cc:1434] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.511s user 0.019s sys 0.003s
W20251214 20:52:22.767529 1206 ts_tablet_manager.cc:1467] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212: Tablet startup took 511ms
W20251214 20:52:22.767654 1206 ts_tablet_manager.cc:1469] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212: Trace:
1214 20:52:13.477526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:13.488332 (+ 10806us) service_pool.cc:225] Handling call
1214 20:52:13.489085 (+ 753us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1214 20:52:13.489117 (+ 32us) ts_tablet_manager.cc:682] Creating new metadata...
1214 20:52:13.556645 (+ 67528us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:13.726339 (+169694us) tablet_replica.cc:206] Creating consensus instance
1214 20:52:13.727149 (+ 810us) inbound_call.cc:173] Queueing success response
1214 20:52:22.125696 (+8398547us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1214 20:52:22.256249 (+130553us) ts_tablet_manager.cc:1435] Starting tablet replica
1214 20:52:22.256457 (+ 208us) tablet_replica.cc:261] Starting instrumentation
1214 20:52:22.257041 (+ 584us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":55,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":68,"spinlock_wait_cycles":1534848,"tablet-open.queue_time_us":8398571}
I20251214 20:52:22.763944 1982 raft_consensus.cc:2243] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.769521 1982 raft_consensus.cc:2272] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.770123 1206 tablet_bootstrap.cc:492] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:22.771865 1206 tablet_bootstrap.cc:654] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.784914 1202 tablet_bootstrap.cc:492] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:22.786260 1202 ts_tablet_manager.cc:1403] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.241s user 0.013s sys 0.025s
I20251214 20:52:22.787197 1202 raft_consensus.cc:359] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.791189 1202 raft_consensus.cc:385] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.791324 1202 raft_consensus.cc:740] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.791586 1202 consensus_queue.cc:260] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.794005 1202 ts_tablet_manager.cc:1434] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251214 20:52:22.766157 2038 consensus_queue.cc:1048] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:22.794317 1202 tablet_bootstrap.cc:492] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:22.786604 2079 raft_consensus.cc:493] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.794574 2079 raft_consensus.cc:515] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.795258 2079 leader_election.cc:290] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.795967 1202 tablet_bootstrap.cc:654] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.797348 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet bc73d590531c409e9f60670435dc8223
I20251214 20:52:22.798168 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_2718b41c3d5643439f3a0d867fbb8116" } new_table_name: "default.Table_a79a6c9f5a07476a84b34893480c5601" modify_external_catalogs: true
I20251214 20:52:22.798694 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bd5c18696f54ea190c1ef84b93c4f63" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:22.798875 915 raft_consensus.cc:2468] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:22.799939 991 leader_election.cc:304] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.800208 891 tablet_service.cc:1552] Processing DeleteTablet for tablet bc73d590531c409e9f60670435dc8223 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.800400 2079 raft_consensus.cc:2804] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:22.800480 2079 raft_consensus.cc:493] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:22.800544 2079 raft_consensus.cc:3060] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:22.800881 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet bc73d590531c409e9f60670435dc8223 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc73d590531c409e9f60670435dc8223 already in progress: creating tablet
I20251214 20:52:22.804615 1202 tablet_bootstrap.cc:492] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:22.804762 1202 ts_tablet_manager.cc:1403] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20251214 20:52:22.805434 1202 raft_consensus.cc:359] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.805577 1202 raft_consensus.cc:385] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.805616 1202 raft_consensus.cc:740] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.805794 1202 consensus_queue.cc:260] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.806128 2038 raft_consensus.cc:493] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.806284 2038 raft_consensus.cc:515] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.807022 2038 leader_election.cc:290] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.808462 1202 ts_tablet_manager.cc:1434] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:22.809180 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e5e220d083243eb8aa4328f366cd7ff" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:22.809376 915 raft_consensus.cc:2468] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:22.810065 991 leader_election.cc:304] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.810423 2038 raft_consensus.cc:2804] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:22.810501 2038 raft_consensus.cc:493] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:22.810561 2038 raft_consensus.cc:3060] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.811009 1202 tablet_bootstrap.cc:492] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:22.812546 1202 tablet_bootstrap.cc:654] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.813431 2038 raft_consensus.cc:515] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.814587 448 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_76f578444d71410a93c3f78ff50b7951 [id=6157817e96d642d09f9941ad7f7e82ee] alter complete (version 2)
I20251214 20:52:22.814654 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ded13abcaf9545afad1de571fd382831"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.815487 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ded13abcaf9545afad1de571fd382831 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.817039 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet f342dafae1d04be3ba38f7c0e4f5d058
I20251214 20:52:22.817912 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_26795454cb054bd6bdb4100765bd3c17" } new_table_name: "default.Table_121c97f6b224464087b06c6fd31d4fa4" modify_external_catalogs: true
I20251214 20:52:22.817813 2079 raft_consensus.cc:515] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.819793 891 tablet_service.cc:1552] Processing DeleteTablet for tablet f342dafae1d04be3ba38f7c0e4f5d058 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:22.820623 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet f342dafae1d04be3ba38f7c0e4f5d058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f342dafae1d04be3ba38f7c0e4f5d058 already in progress: creating tablet
I20251214 20:52:22.820684 1202 tablet_bootstrap.cc:492] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:22.820828 1202 ts_tablet_manager.cc:1403] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20251214 20:52:22.821484 1202 raft_consensus.cc:359] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.821553 2079 leader_election.cc:290] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.821620 1202 raft_consensus.cc:385] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.821667 1202 raft_consensus.cc:740] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.822000 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_29dd5d466c7d4c1698b88311145db749" } new_table_name: "default.Table_8a02e24feeab4dc6a69f6ca246440bad" modify_external_catalogs: true
I20251214 20:52:22.822548 2038 leader_election.cc:290] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.822187 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bd5c18696f54ea190c1ef84b93c4f63" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:22.823097 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e5e220d083243eb8aa4328f366cd7ff" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:22.823271 909 raft_consensus.cc:3060] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.823294 915 raft_consensus.cc:3060] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:22.826690 909 raft_consensus.cc:2468] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:22.826928 915 raft_consensus.cc:2468] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:22.827193 1202 consensus_queue.cc:260] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.827697 1202 ts_tablet_manager.cc:1434] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:22.827994 1202 tablet_bootstrap.cc:492] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:22.827936 991 leader_election.cc:304] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.828274 991 leader_election.cc:304] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:22.832296 1202 tablet_bootstrap.cc:654] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.838832 1202 tablet_bootstrap.cc:492] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:22.838984 1202 ts_tablet_manager.cc:1403] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251214 20:52:22.839764 1202 raft_consensus.cc:359] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.839946 1202 raft_consensus.cc:385] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.840005 1202 raft_consensus.cc:740] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.840225 1202 consensus_queue.cc:260] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.840773 1202 ts_tablet_manager.cc:1434] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:22.841038 1202 tablet_bootstrap.cc:492] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:22.841666 454 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 566c6e4761d5486d964e276d151e32b0 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 566c6e4761d5486d964e276d151e32b0 consensus state has no leader
I20251214 20:52:22.843232 1202 tablet_bootstrap.cc:654] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.844079 2038 raft_consensus.cc:2804] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:22.844300 1650 raft_consensus.cc:493] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.844419 1650 raft_consensus.cc:515] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:22.844997 1650 leader_election.cc:290] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:22.851471 1650 raft_consensus.cc:493] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.851609 1650 raft_consensus.cc:515] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.852192 1650 leader_election.cc:290] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.861375 2167 raft_consensus.cc:2804] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:22.872421 2167 raft_consensus.cc:697] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:22.872967 2167 consensus_queue.cc:237] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.889506 891 tablet_service.cc:1505] Processing CreateTablet for tablet 49778cfe80f6483389cfd421abee8ca7 (DEFAULT_TABLE table=default.Table_3024ac839d9441db89239acebb00bc12 [id=d3229230c8844d2fb42cd81149462a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.890101 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49778cfe80f6483389cfd421abee8ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.779747 1982 ts_tablet_manager.cc:1916] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:22.791108 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 109711us
I20251214 20:52:22.845904 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18e119bc44874aecb4e83c8d6040c8cd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:22.899019 454 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 4dcd511a5dca4a2cbda81fbc255ce354 consensus state has no leader
I20251214 20:52:22.853166 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca263a80702f4f64aa7c872c2be08207" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:22.850423 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 604b7c6fe3b24daca70711a1990af556 (table default.Table_fbb5b21626c142c8baac60a7f29d3c51 [id=7cb78d704474480db14d212ee89166a0]) successfully deleted
W20251214 20:52:22.876222 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 825baba561ee4a8ebd44f75d8c51706a consensus state has no leader
W20251214 20:52:22.900075 848 leader_election.cc:343] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:22.900331 848 leader_election.cc:343] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:22.900672 713 leader_election.cc:343] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: c16895ca08d94b56b5c4bc9352e330ef
I20251214 20:52:22.900794 713 leader_election.cc:304] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:22.900727 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_309db467eb00418598443e6f7a007e44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:22.901111 1964 raft_consensus.cc:2749] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 20:52:22.901422 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_309db467eb00418598443e6f7a007e44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:22.876518 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6ebfc373fbf41b1920c271a15302484 consensus state has no leader
I20251214 20:52:22.902937 450 master_service.cc:954] ReplaceTablet: received request to replace tablet eb29fd8c85344ed8aa8d880e2ed240bc from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:22.903548 751 tablet_service.cc:1505] Processing CreateTablet for tablet 49778cfe80f6483389cfd421abee8ca7 (DEFAULT_TABLE table=default.Table_3024ac839d9441db89239acebb00bc12 [id=d3229230c8844d2fb42cd81149462a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.904001 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49778cfe80f6483389cfd421abee8ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.905280 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet a10663bd306d40e7a6375d06d69be22d (table default.Table_f93eb4d354254ec79674c615721834b7 [id=e8e0419a222b460ab608def1752925f0]) successfully deleted
W20251214 20:52:22.905493 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 9b1a937150ce4ecfa210c8465d682496 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b1a937150ce4ecfa210c8465d682496 already in progress: creating tablet
I20251214 20:52:22.906869 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.907168 738 tablet_service.cc:1552] Processing DeleteTablet for tablet f342dafae1d04be3ba38f7c0e4f5d058 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.907438 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f4760585baf4b4ab3b9b94b5a748a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.907672 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b47ca1a03944cf1847d3578856dafa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.907910 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 274185a4115b458e8e9a376bdbcc870b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.908093 744 tablet_service.cc:1552] Processing DeleteTablet for tablet bc73d590531c409e9f60670435dc8223 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.877679 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_8ae98ffc7458410786e7844521f3c5c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.908744 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ae98ffc7458410786e7844521f3c5c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.879292 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_fc742d049909446bbdb1a0b0973734c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.910341 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc742d049909446bbdb1a0b0973734c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:22.911536 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 61144.8us
I20251214 20:52:22.917403 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bd5c18696f54ea190c1ef84b93c4f63" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:22.918320 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e5e220d083243eb8aa4328f366cd7ff" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:22.918957 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3e5e220d083243eb8aa4328f366cd7ff" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:22.919588 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bd5c18696f54ea190c1ef84b93c4f63" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:22.931799 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:22.932788 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2842f1dc0c374d24946f90dac1cf8aa5 because the tablet was not found. No further retry: Not found: Tablet not found: 2842f1dc0c374d24946f90dac1cf8aa5
I20251214 20:52:22.935313 1203 tablet_bootstrap.cc:492] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:22.935482 1203 ts_tablet_manager.cc:1403] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.327s user 0.025s sys 0.049s
I20251214 20:52:22.936199 1203 raft_consensus.cc:359] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.936364 1203 raft_consensus.cc:385] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.936408 1203 raft_consensus.cc:740] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.936580 1203 consensus_queue.cc:260] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:22.936674 454 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet d171301941fa43908e05984c3b016007 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet d171301941fa43908e05984c3b016007 consensus state has no leader
I20251214 20:52:22.937064 1203 ts_tablet_manager.cc:1434] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:22.937345 1203 tablet_bootstrap.cc:492] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:22.938755 1203 tablet_bootstrap.cc:654] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:22.939882 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b1a937150ce4ecfa210c8465d682496 with delete_type TABLET_DATA_DELETED (replaced by tablet f1eb4c3aa654480caaa69dbd5047365c) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.941480 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet eb29fd8c85344ed8aa8d880e2ed240bc
I20251214 20:52:22.942039 450 catalog_manager.cc:6724] ReplaceTablet: tablet eb29fd8c85344ed8aa8d880e2ed240bc deleted and replaced by tablet 7242be189ca54c9182181e33d6743e6b
I20251214 20:52:22.942366 450 master_service.cc:954] ReplaceTablet: received request to replace tablet 35a529702f924a5c9bf53c3c668c6094 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:22.943035 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ee3a88061eb34c19b3fad30276bc7ece"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:22.944041 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee3a88061eb34c19b3fad30276bc7ece in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.945585 1206 tablet_bootstrap.cc:492] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:22.945766 1206 ts_tablet_manager.cc:1403] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.177s user 0.004s sys 0.000s
I20251214 20:52:22.946609 1206 raft_consensus.cc:359] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.946878 1206 raft_consensus.cc:385] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:22.946851 744 tablet_service.cc:1505] Processing CreateTablet for tablet a4643effa71d4d18850913a59ae1c75d (DEFAULT_TABLE table=default.Table_bb0aa14e61414a5c95c2923827b7cae7 [id=91f3b4dd176348a5aa38c6bdc417918f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:22.946970 1206 raft_consensus.cc:740] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:22.947333 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4643effa71d4d18850913a59ae1c75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:22.949255 453 master_service.cc:954] ReplaceTablet: received request to replace tablet 023299180d074893847e84dcd9812751 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:22.949775 751 tablet_service.cc:1552] Processing DeleteTablet for tablet eb29fd8c85344ed8aa8d880e2ed240bc with delete_type TABLET_DATA_DELETED (replaced by tablet 7242be189ca54c9182181e33d6743e6b) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:22.951601 891 tablet_service.cc:1552] Processing DeleteTablet for tablet eb29fd8c85344ed8aa8d880e2ed240bc with delete_type TABLET_DATA_DELETED (replaced by tablet 7242be189ca54c9182181e33d6743e6b) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:22.952412 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet eb29fd8c85344ed8aa8d880e2ed240bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb29fd8c85344ed8aa8d880e2ed240bc already in progress: creating tablet
I20251214 20:52:22.954773 2038 raft_consensus.cc:697] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:22.955214 2038 consensus_queue.cc:237] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.955727 1964 raft_consensus.cc:493] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.955901 1964 raft_consensus.cc:515] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.956858 1964 leader_election.cc:290] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:22.957897 712 leader_election.cc:343] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 604b7c6fe3b24daca70711a1990af556
W20251214 20:52:22.958889 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 130600us
I20251214 20:52:22.960202 1650 raft_consensus.cc:493] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:22.960366 1650 raft_consensus.cc:515] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.960985 1650 leader_election.cc:290] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:22.962834 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2e83bca8ed642b6a1b705ef70fde27f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:22.963469 848 leader_election.cc:343] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.970247 1982 ts_tablet_manager.cc:1929] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:22.970428 1982 log.cc:1199] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/24ff58f321124d12b5330dca185d426e
I20251214 20:52:22.970715 1982 ts_tablet_manager.cc:1950] T 24ff58f321124d12b5330dca185d426e P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:22.971026 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 112) took 2744 ms. Trace:
I20251214 20:52:22.974694 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.974901 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff with delete_type TABLET_DATA_DELETED (replaced by tablet 6a000951f61948ac85fdd431f3d162c2) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:22.976354 989 leader_election.cc:343] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.977077 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cafed2861e94cd9b0d10d33864ba45d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.977340 1977 tablet_replica.cc:333] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:22.977526 1977 raft_consensus.cc:2243] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:22.977717 1977 raft_consensus.cc:2272] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:22.978406 1977 ts_tablet_manager.cc:1916] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:22.947228 1206 consensus_queue.cc:260] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:22.984068 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af99e0bd8774a678575847a9ec2e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:22.984752 989 leader_election.cc:343] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:22.985209 989 leader_election.cc:343] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:22.985486 989 leader_election.cc:343] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:22.987485 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 16816fa6b8c34207bb929147910b93cb with delete_type TABLET_DATA_DELETED (replaced by tablet 532c44785f204f988518c6fd628f76ca) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.988809 883 tablet_service.cc:1552] Processing DeleteTablet for tablet b315744c8cd043c5a8bdceec73a7e28b with delete_type TABLET_DATA_DELETED (replaced by tablet f945814952b14131ba4e0c67a403a680) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.989130 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ac06f852fc4abea5cb213926002ca1 with delete_type TABLET_DATA_DELETED (replaced by tablet 248e3666fd13434e97378eb5a98dfadd) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.983402 2168 consensus_queue.cc:1048] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20251214 20:52:22.989691 890 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24e4f4e74040249068505058c15622 with delete_type TABLET_DATA_DELETED (replaced by tablet 0dbfe8ee68bf4a2c88489b1f81b36991) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.989945 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 60ae3db968f6412aa0c3952fafe65f88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.990298 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 2da66ec83e9140f7a605f0c86abff468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.990689 883 tablet_service.cc:1552] Processing DeleteTablet for tablet a063083f54ad49d2a842fa080d44ed85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.990909 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d164747cb294bd4b126b4420231867d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.991297 890 tablet_service.cc:1552] Processing DeleteTablet for tablet ca263a80702f4f64aa7c872c2be08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.991634 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 2a7fc15882e2474bb6885070be13b1fa with delete_type TABLET_DATA_DELETED (replaced by tablet 65e79db74fa046698507c83416ee1669) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.991835 890 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b22043d81b493fb12116422b1d3fb1 with delete_type TABLET_DATA_DELETED (replaced by tablet ba7952290e2c4746802cffeb14971da0) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.992702 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 012b81223d0f4ae9b742014e329f11a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.993057 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.993397 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b1a937150ce4ecfa210c8465d682496 with delete_type TABLET_DATA_DELETED (replaced by tablet f1eb4c3aa654480caaa69dbd5047365c) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.993726 1203 tablet_bootstrap.cc:492] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:22.993739 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 06381efe75284e649fd6d1a9903cfa9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.993880 1203 ts_tablet_manager.cc:1403] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.057s user 0.000s sys 0.011s
I20251214 20:52:22.994148 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 6381cbf05a8c4f84b71f229497fd28ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:10 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:22.994470 991 consensus_peers.cc:597] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e16fe751f10f4616a00d0af811998729. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:22.994498 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.994781 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b86f4adaf4540f48a4f42bc45e259e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.995152 890 tablet_service.cc:1552] Processing DeleteTablet for tablet ba50e32f7e724868bb891045a939e362 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.995527 890 tablet_service.cc:1552] Processing DeleteTablet for tablet c9b7149a436a4cab80b39adccf8567f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.996929 1977 ts_tablet_manager.cc:1929] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:22.997128 1977 log.cc:1199] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/9c0f49e5d78240a296e47eccb3218e4f
I20251214 20:52:22.997586 1977 ts_tablet_manager.cc:1950] T 9c0f49e5d78240a296e47eccb3218e4f P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:22.999027 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 9c0f49e5d78240a296e47eccb3218e4f (table default.Table_ec65ee1eda49435895dab50f78b4ee15 [id=8a0182244a254993998d4ea00d03cd9d]) successfully deleted
I20251214 20:52:22.999392 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 2842f1dc0c374d24946f90dac1cf8aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:11 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:22.999745 1977 tablet_replica.cc:333] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:23.000026 1977 raft_consensus.cc:2243] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.000237 1977 raft_consensus.cc:2272] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.000547 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 274185a4115b458e8e9a376bdbcc870b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.000783 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f4760585baf4b4ab3b9b94b5a748a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.001278 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca263a80702f4f64aa7c872c2be08207" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:23.002053 851 leader_election.cc:343] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.002178 851 leader_election.cc:304] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:23.002566 1650 raft_consensus.cc:2749] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:23.003214 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3962c33a294fa98f8bd485268aee71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.003561 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b47ca1a03944cf1847d3578856dafa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.004263 1977 ts_tablet_manager.cc:1916] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.004549 891 tablet_service.cc:1552] Processing DeleteTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.004606 1856 ts_tablet_manager.cc:1929] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:23.004729 1856 log.cc:1199] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/3bbb28259c1442ae94f4eff76c25c539
I20251214 20:52:23.005030 1856 ts_tablet_manager.cc:1950] T 3bbb28259c1442ae94f4eff76c25c539 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:23.002713 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2e83bca8ed642b6a1b705ef70fde27f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:23.006013 851 leader_election.cc:343] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.006150 851 leader_election.cc:304] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:23.006498 1650 raft_consensus.cc:2749] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:23.008311 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18e119bc44874aecb4e83c8d6040c8cd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:23.008545 1050 raft_consensus.cc:2468] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:23.009263 1203 raft_consensus.cc:359] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.009413 1203 raft_consensus.cc:385] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.009457 1203 raft_consensus.cc:740] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.009577 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 140) took 2131 ms. Trace:
I20251214 20:52:23.009614 1203 consensus_queue.cc:260] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.009640 1856 rpcz_store.cc:276] 1214 20:52:20.877823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.878163 (+ 340us) service_pool.cc:225] Handling call
1214 20:52:22.763018 (+1884855us) ts_tablet_manager.cc:1151] Deleting tablet 3bbb28259c1442ae94f4eff76c25c539
1214 20:52:22.763033 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.990253 (+227220us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.004494 (+ 14241us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.009562 (+ 5068us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884654}
I20251214 20:52:23.009986 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 141) took 2131 ms. Trace:
I20251214 20:52:23.010046 1203 ts_tablet_manager.cc:1434] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251214 20:52:23.010049 1856 rpcz_store.cc:276] 1214 20:52:20.878265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.878396 (+ 131us) service_pool.cc:225] Handling call
1214 20:52:23.009898 (+2131502us) ts_tablet_manager.cc:1151] Deleting tablet 9c0f49e5d78240a296e47eccb3218e4f
1214 20:52:23.009907 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.009973 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2131311}
I20251214 20:52:23.010337 1203 tablet_bootstrap.cc:492] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:23.010406 1856 tablet_replica.cc:333] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:23.010551 1856 raft_consensus.cc:2243] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:23.010728 1856 pending_rounds.cc:70] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:23.010775 1856 pending_rounds.cc:77] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493743610843136 op_type: NO_OP noop_request { }
I20251214 20:52:23.010849 1856 raft_consensus.cc:2889] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:23.010916 1856 raft_consensus.cc:2272] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.010972 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e256d4537249a394b80dc36beefef4 with delete_type TABLET_DATA_DELETED (replaced by tablet cea9423c40044e4cabb8bbd755a46688) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.011672 1203 tablet_bootstrap.cc:654] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.011785 891 tablet_service.cc:1505] Processing CreateTablet for tablet a4643effa71d4d18850913a59ae1c75d (DEFAULT_TABLE table=default.Table_bb0aa14e61414a5c95c2923827b7cae7 [id=91f3b4dd176348a5aa38c6bdc417918f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.012143 851 leader_election.cc:304] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:23.013068 1650 raft_consensus.cc:2804] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:23.013091 1856 ts_tablet_manager.cc:1916] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.013139 1650 raft_consensus.cc:493] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:23.013200 1650 raft_consensus.cc:3060] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.015113 2168 consensus_queue.cc:1048] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:23.015974 1650 raft_consensus.cc:515] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:23.017697 1650 leader_election.cc:290] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:23.012607 890 tablet_service.cc:1552] Processing DeleteTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.019534 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18e119bc44874aecb4e83c8d6040c8cd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:23.019719 1050 raft_consensus.cc:3060] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.020606 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 9c0f49e5d78240a296e47eccb3218e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.021256 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.021589 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 52daf1204bf142e7b82852bdec6582fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.021939 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 472e4a5990e64d02b1d52dc6e86d7190 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:23.022083 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:34819, user_credentials={real_user=slave}} blocked reactor thread for 109528us
I20251214 20:52:23.022229 1050 raft_consensus.cc:2468] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:23.022383 890 tablet_service.cc:1552] Processing DeleteTablet for tablet cd88d1e95ee04ef596863e1ae3687de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.022782 890 tablet_service.cc:1552] Processing DeleteTablet for tablet bb698ff30fcc47d09000cc68d2c5225b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.023160 851 leader_election.cc:304] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:23.023522 1650 raft_consensus.cc:2804] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:23.023649 1650 raft_consensus.cc:697] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:23.024000 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 16816fa6b8c34207bb929147910b93cb with delete_type TABLET_DATA_DELETED (replaced by tablet 532c44785f204f988518c6fd628f76ca) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.024334 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 49778cfe80f6483389cfd421abee8ca7 (DEFAULT_TABLE table=default.Table_3024ac839d9441db89239acebb00bc12 [id=d3229230c8844d2fb42cd81149462a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.024423 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet bc73d590531c409e9f60670435dc8223 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.024446 1024 tablet_service.cc:1505] Processing CreateTablet for tablet a4643effa71d4d18850913a59ae1c75d (DEFAULT_TABLE table=default.Table_bb0aa14e61414a5c95c2923827b7cae7 [id=91f3b4dd176348a5aa38c6bdc417918f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.024714 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49778cfe80f6483389cfd421abee8ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.024878 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4643effa71d4d18850913a59ae1c75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.025213 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet c8e33ba1149a44c294885d76f676160a with delete_type TABLET_DATA_DELETED (replaced by tablet 1eff8a23d48c43e2bdbfb34f4bab6edf) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.025369 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dca369dc5f54a4ea1783e334c5670d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.025491 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 762068d560ff47918dd9fd8791083433 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.025538 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 3bbb28259c1442ae94f4eff76c25c539 (table default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]) successfully deleted
I20251214 20:52:23.025601 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cafed2861e94cd9b0d10d33864ba45d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.025719 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d7c7b3f62764ff497f4d70b53eed7de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:23.025797 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 9c0f49e5d78240a296e47eccb3218e4f because the tablet was not found. No further retry: Not found: Tablet not found: 9c0f49e5d78240a296e47eccb3218e4f
I20251214 20:52:23.026639 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet f342dafae1d04be3ba38f7c0e4f5d058 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.027077 1203 tablet_bootstrap.cc:492] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:23.027220 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet eb29fd8c85344ed8aa8d880e2ed240bc with delete_type TABLET_DATA_DELETED (replaced by tablet 7242be189ca54c9182181e33d6743e6b) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.027245 1203 ts_tablet_manager.cc:1403] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20251214 20:52:23.027424 1650 consensus_queue.cc:237] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:23.028079 1203 raft_consensus.cc:359] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.028245 1203 raft_consensus.cc:385] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.028290 1203 raft_consensus.cc:740] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.028488 1203 consensus_queue.cc:260] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.029024 1203 ts_tablet_manager.cc:1434] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:23.029317 1203 tablet_bootstrap.cc:492] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:23.030788 1203 tablet_bootstrap.cc:654] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.032502 1856 ts_tablet_manager.cc:1929] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.032614 1856 log.cc:1199] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/3e5e220d083243eb8aa4328f366cd7ff
I20251214 20:52:23.034667 890 tablet_service.cc:1552] Processing DeleteTablet for tablet a10663bd306d40e7a6375d06d69be22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.035360 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dca369dc5f54a4ea1783e334c5670d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.036691 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 4f22ac24c38e4c64bb26c54ae4d87dfb (DEFAULT_TABLE table=default.Table_b2ac4ca26b6640a287e7f3b6974c2363 [id=15ed55ac8a0249bfbc08076265293bd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.037223 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f22ac24c38e4c64bb26c54ae4d87dfb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:22.886353 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:46855 (request call id 163) took 1631 ms (client timeout 1000 ms). Trace:
W20251214 20:52:23.037695 446 rpcz_store.cc:269] 1214 20:52:21.254974 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.300889 (+ 45915us) service_pool.cc:225] Handling call
1214 20:52:22.886290 (+1585401us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.041199 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.041350 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.041357 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:22.041360 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.041552 (+ 192us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.041571 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.041577 (+ 6us) tablet.cc:683] Acquiring locks for 3 operations
1214 20:52:22.041693 (+ 116us) tablet.cc:699] Row locks acquired
1214 20:52:22.041696 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.041743 (+ 47us) write_op.cc:270] Start()
1214 20:52:22.041786 (+ 43us) write_op.cc:276] Timestamp: P: 1765745542041631 usec, L: 0
1214 20:52:22.041790 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.097391 (+ 55601us) log.cc:844] Serialized 1275 byte log entry
1214 20:52:22.103416 (+ 6025us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 000000000098492a 00007ff413306922 00007ff4146e9875 00007ff4146e8399 00007ff4146f07c0 00007ff4146f05cd 0000000000a8970c 00007ff40748e20f 00007ff40749152b 00007ff4074913a8 0000000000a8970c
1214 20:52:22.106148 (+ 2732us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.106307 (+ 159us) write_op.cc:301] APPLY: starting
1214 20:52:22.106328 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.106396 (+ 68us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.106399 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.106519 (+ 120us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.106563 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1214 20:52:22.106572 (+ 9us) write_op.cc:312] APPLY: finished
1214 20:52:22.106671 (+ 99us) log.cc:844] Serialized 35 byte log entry
1214 20:52:22.106939 (+ 268us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.106982 (+ 43us) write_op.cc:454] Released schema lock
1214 20:52:22.106991 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.108740 (+ 1749us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.108746 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
1214 20:52:22.276209 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.276317 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.276323 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:22.276325 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:22.276443 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.276454 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.276459 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.276525 (+ 66us) tablet.cc:699] Row locks acquired
1214 20:52:22.276527 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:22.276565 (+ 38us) write_op.cc:270] Start()
1214 20:52:22.276601 (+ 36us) write_op.cc:276] Timestamp: P: 1765745542276512 usec, L: 0
1214 20:52:22.276605 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.276748 (+ 143us) log.cc:844] Serialized 356 byte log entry
1214 20:52:22.289128 (+ 12380us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.289202 (+ 74us) write_op.cc:301] APPLY: starting
1214 20:52:22.289213 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.289240 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.289262 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.289295 (+ 33us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.289318 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.289324 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:22.289392 (+ 68us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.289443 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.289457 (+ 14us) write_op.cc:454] Released schema lock
1214 20:52:22.289463 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.306881 (+ 17418us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.306890 (+ 9us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":51445,"reported_tablets":152,"child_traces":[["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":582,"apply.run_wall_time_us":2416,"num_ops":3,"prepare.queue_time_us":47828,"prepare.run_cpu_time_us":1424,"prepare.run_wall_time_us":64236,"raft.queue_time_us":85,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":1773,"replication_time_us":64326,"spinlock_wait_cycles":127094528}],["op",{"apply.queue_time_us":36,"apply.run_cpu_time_us":309,"apply.run_wall_time_us":310,"num_ops":1,"prepare.queue_time_us":24371,"prepare.run_cpu_time_us":7788,"prepare.run_wall_time_us":30642,"raft.queue_time_us":961,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":12503,"thread_start_us":24183,"threads_started":1}]]}
W20251214 20:52:22.938741 454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:39607 (request call id 154) took 886 ms (client timeout 1000 ms). Trace:
W20251214 20:52:23.039781 454 rpcz_store.cc:269] 1214 20:52:22.052747 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.120636 (+ 67889us) service_pool.cc:225] Handling call
1214 20:52:22.938678 (+818042us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":167}
I20251214 20:52:23.040103 1856 ts_tablet_manager.cc:1950] T 3e5e220d083243eb8aa4328f366cd7ff P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:23.041445 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 143) took 2162 ms. Trace:
I20251214 20:52:23.042016 1856 rpcz_store.cc:276] 1214 20:52:20.878574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.878623 (+ 49us) service_pool.cc:225] Handling call
1214 20:52:23.010368 (+2131745us) ts_tablet_manager.cc:1151] Deleting tablet 3e5e220d083243eb8aa4328f366cd7ff
1214 20:52:23.010377 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.016590 (+ 6213us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.032397 (+ 15807us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.041426 (+ 9029us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2130568}
I20251214 20:52:23.042932 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 144) took 2164 ms. Trace:
I20251214 20:52:23.043318 1856 rpcz_store.cc:276] 1214 20:52:20.878711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.879946 (+ 1235us) service_pool.cc:225] Handling call
1214 20:52:23.042834 (+2162888us) ts_tablet_manager.cc:1151] Deleting tablet 8af99e0bd8774a678575847a9ec2e557
1214 20:52:23.042845 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.042919 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2162672}
I20251214 20:52:23.044410 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 145) took 2164 ms. Trace:
I20251214 20:52:23.044768 1856 rpcz_store.cc:276] 1214 20:52:20.879557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.879612 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:23.044318 (+2164706us) ts_tablet_manager.cc:1151] Deleting tablet 52daf1204bf142e7b82852bdec6582fb
1214 20:52:23.044328 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.044397 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2164105}
I20251214 20:52:23.045290 890 tablet_service.cc:1552] Processing DeleteTablet for tablet e16fe751f10f4616a00d0af811998729 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.045828 890 tablet_service.cc:1552] Processing DeleteTablet for tablet ec85857f047e44d78835305ddcb70768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.046072 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c973e0111804b8b869068bdcc5869bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.046407 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 762068d560ff47918dd9fd8791083433 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.046763 885 tablet_service.cc:1552] Processing DeleteTablet for tablet dfe50a6219134431b0a025317932609b with delete_type TABLET_DATA_DELETED (replaced by tablet 6b482043af42485eb4c73d6eb864a963) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.047073 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 2accf8a160584c82bb695a52e3f2b05c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.047427 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.047940 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 6eca34627bbc4a3f9915be72f3128370 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d535d8f71b44c81bbba6937e804d900) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.048245 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d6e7306254044449f1127fb963ca5f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.048561 885 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.049158 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_e125108df3c94234bedbdf467db81f28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.052104 2143 raft_consensus.cc:493] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.052887 2143 raft_consensus.cc:515] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.053202 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 292a28b12df246a8bc855d81c530746e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.053588 2143 leader_election.cc:290] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:23.054145 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4643effa71d4d18850913a59ae1c75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.055373 885 tablet_service.cc:1552] Processing DeleteTablet for tablet c8e33ba1149a44c294885d76f676160a with delete_type TABLET_DATA_DELETED (replaced by tablet 1eff8a23d48c43e2bdbfb34f4bab6edf) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:23.055526 848 consensus_peers.cc:597] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:23.056015 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 8eb1006b0ba440dcba2f9de04dca2702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.056778 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d7c7b3f62764ff497f4d70b53eed7de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.057291 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.053969 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7dd657f3baff40bb8cb7f4648367f7a1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:23.059226 1050 raft_consensus.cc:2468] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:23.055508 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 54e5fff1fc84419ab3d3131cb3a12d92 (DEFAULT_TABLE table=default.Table_1e0e46b1f1e1418eadafc528585358f3 [id=d99b78f1c7dd40af85d84125b089518e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.059463 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 17550eb0ee6b476f8a8c304451f678b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:23.059780 455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:39607 (request call id 153) took 2071 ms (client timeout 1000 ms). Trace:
I20251214 20:52:23.059816 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e5fff1fc84419ab3d3131cb3a12d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.059973 851 leader_election.cc:304] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:23.062100 2143 raft_consensus.cc:2804] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:23.062219 2143 raft_consensus.cc:493] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:23.062299 2143 raft_consensus.cc:3060] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:22.973821 712 consensus_peers.cc:597] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fcfcfe81fe24d369eea696efc61d21c. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:23.002864 713 leader_election.cc:343] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 604b7c6fe3b24daca70711a1990af556
W20251214 20:52:23.062539 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 88748.3us
I20251214 20:52:23.062566 713 leader_election.cc:304] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:23.062820 2155 raft_consensus.cc:2749] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:23.063112 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 60255.5us
I20251214 20:52:23.063840 751 tablet_service.cc:1505] Processing CreateTablet for tablet 54e5fff1fc84419ab3d3131cb3a12d92 (DEFAULT_TABLE table=default.Table_1e0e46b1f1e1418eadafc528585358f3 [id=d99b78f1c7dd40af85d84125b089518e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.064811 2143 raft_consensus.cc:515] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.064976 2155 consensus_queue.cc:1048] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:23.065927 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 142) took 2187 ms. Trace:
I20251214 20:52:23.065912 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7dd657f3baff40bb8cb7f4648367f7a1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:23.065994 1856 rpcz_store.cc:276] 1214 20:52:20.878406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.879834 (+ 1428us) service_pool.cc:225] Handling call
1214 20:52:23.065823 (+2185989us) ts_tablet_manager.cc:1151] Deleting tablet c2ae9dc8ead8403cb7215af8778872f0
1214 20:52:23.065844 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.065914 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2185526}
I20251214 20:52:23.018174 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18e119bc44874aecb4e83c8d6040c8cd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:23.066239 1856 tablet_replica.cc:333] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:23.035403 1964 raft_consensus.cc:493] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:23.066442 848 leader_election.cc:343] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.066565 1964 raft_consensus.cc:515] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.054301 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7dd657f3baff40bb8cb7f4648367f7a1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:23.067070 848 leader_election.cc:343] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.067196 1964 leader_election.cc:290] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:22.980170 1206 ts_tablet_manager.cc:1434] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
W20251214 20:52:23.067541 848 leader_election.cc:343] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.067770 1206 tablet_bootstrap.cc:492] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:23.068410 2143 leader_election.cc:290] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:23.070047 1202 tablet_bootstrap.cc:492] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:23.070281 1202 ts_tablet_manager.cc:1403] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.229s user 0.015s sys 0.025s
I20251214 20:52:23.070971 1206 tablet_bootstrap.cc:654] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.071539 744 tablet_service.cc:1505] Processing CreateTablet for tablet 4f22ac24c38e4c64bb26c54ae4d87dfb (DEFAULT_TABLE table=default.Table_b2ac4ca26b6640a287e7f3b6974c2363 [id=15ed55ac8a0249bfbc08076265293bd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.072561 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e5fff1fc84419ab3d3131cb3a12d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.072768 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f22ac24c38e4c64bb26c54ae4d87dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.077813 1202 raft_consensus.cc:359] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.078042 1202 raft_consensus.cc:385] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.078110 1202 raft_consensus.cc:740] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.079622 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7d709316b24b8eb223a838710c2979" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:23.051689 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 3e5e220d083243eb8aa4328f366cd7ff (table default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]) successfully deleted
W20251214 20:52:23.080348 991 consensus_peers.cc:597] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02e256d4537249a394b80dc36beefef4. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:23.061604 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e125108df3c94234bedbdf467db81f28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:22.971109 1982 rpcz_store.cc:276] 1214 20:52:20.226358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.226408 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:22.758452 (+2532044us) ts_tablet_manager.cc:1151] Deleting tablet 24ff58f321124d12b5330dca185d426e
1214 20:52:22.758461 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:22.930345 (+171884us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.970117 (+ 39772us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:22.971007 (+ 890us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2384776}
I20251214 20:52:23.082619 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7dd657f3baff40bb8cb7f4648367f7a1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:23.082784 1050 raft_consensus.cc:3060] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.082842 1982 tablet_replica.cc:333] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.083086 1982 raft_consensus.cc:2243] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.083298 1982 raft_consensus.cc:2272] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.083289 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee7d709316b24b8eb223a838710c2979" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:23.083879 712 leader_election.cc:304] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:23.079559 455 rpcz_store.cc:269] 1214 20:52:20.987918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.003497 (+ 15579us) service_pool.cc:225] Handling call
1214 20:52:23.059735 (+2056238us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35474,"reported_tablets":168}
I20251214 20:52:23.083854 751 tablet_service.cc:1505] Processing CreateTablet for tablet 336cb9d03fd44f70930662ee3c8d987a (DEFAULT_TABLE table=default.Table_ebccdf7648384d44b9629a1b41509e68 [id=2e1e99c74ef5467190886342facb1a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.084308 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 336cb9d03fd44f70930662ee3c8d987a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.083557 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 24ff58f321124d12b5330dca185d426e (table default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]) successfully deleted
I20251214 20:52:23.084326 1964 raft_consensus.cc:2749] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:23.080422 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8af99e0bd8774a678575847a9ec2e557 because the tablet was not found. No further retry: Not found: Tablet not found: 8af99e0bd8774a678575847a9ec2e557
W20251214 20:52:23.084805 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 52daf1204bf142e7b82852bdec6582fb because the tablet was not found. No further retry: Not found: Tablet not found: 52daf1204bf142e7b82852bdec6582fb
W20251214 20:52:23.085027 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c2ae9dc8ead8403cb7215af8778872f0 because the tablet was not found. No further retry: Not found: Tablet not found: c2ae9dc8ead8403cb7215af8778872f0
I20251214 20:52:23.085354 1050 raft_consensus.cc:2468] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:23.085974 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 336cb9d03fd44f70930662ee3c8d987a (DEFAULT_TABLE table=default.Table_ebccdf7648384d44b9629a1b41509e68 [id=2e1e99c74ef5467190886342facb1a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.086472 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 336cb9d03fd44f70930662ee3c8d987a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.089146 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_581df19239f343f89fbe7e5c17d6d85c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:23.089859 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_581df19239f343f89fbe7e5c17d6d85c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.090935 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 604b7c6fe3b24daca70711a1990af556 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.093189 961 heartbeater.cc:499] Master 127.0.104.62:42865 was elected leader, sending a full tablet report...
I20251214 20:52:23.093928 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ad73c92d80cf4c5d9f102d239903cb19" } new_table_name: "default.Table_5bdf1b4c267c425ca1678f5de402fc9f" modify_external_catalogs: true
I20251214 20:52:23.095894 851 leader_election.cc:304] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:23.096242 2143 raft_consensus.cc:2804] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:23.099843 1206 tablet_bootstrap.cc:492] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:23.100072 1206 ts_tablet_manager.cc:1403] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
W20251214 20:52:23.105868 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 119967us
I20251214 20:52:23.108124 1203 tablet_bootstrap.cc:492] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:23.108335 1203 ts_tablet_manager.cc:1403] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.001s
W20251214 20:52:23.108471 848 consensus_peers.cc:597] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:23.109280 885 tablet_service.cc:1552] Processing DeleteTablet for tablet ee7d709316b24b8eb223a838710c2979 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.109118 1203 raft_consensus.cc:359] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.109396 1203 raft_consensus.cc:385] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.109448 1203 raft_consensus.cc:740] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.109637 1203 consensus_queue.cc:260] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.110280 885 tablet_service.cc:1505] Processing CreateTablet for tablet 4f22ac24c38e4c64bb26c54ae4d87dfb (DEFAULT_TABLE table=default.Table_b2ac4ca26b6640a287e7f3b6974c2363 [id=15ed55ac8a0249bfbc08076265293bd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.110522 1203 ts_tablet_manager.cc:1434] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:23.110728 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f22ac24c38e4c64bb26c54ae4d87dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.110852 1203 tablet_bootstrap.cc:492] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:23.112591 1203 tablet_bootstrap.cc:654] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:23.115633 449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:50727 (request call id 183) took 2069 ms (client timeout 1000 ms). Trace:
I20251214 20:52:23.078367 1202 consensus_queue.cc:260] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:23.115756 449 rpcz_store.cc:269] 1214 20:52:21.046311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.065401 (+ 19090us) service_pool.cc:225] Handling call
1214 20:52:23.115556 (+2050155us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:21.879244 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:21.879387 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:21.879394 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:21.879396 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:21.879552 (+ 156us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:21.879567 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:21.879571 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:21.879643 (+ 72us) tablet.cc:699] Row locks acquired
1214 20:52:21.879645 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:21.879700 (+ 55us) write_op.cc:270] Start()
1214 20:52:21.879739 (+ 39us) write_op.cc:276] Timestamp: P: 1765745541879611 usec, L: 0
1214 20:52:21.879743 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:21.879971 (+ 228us) log.cc:844] Serialized 356 byte log entry
1214 20:52:21.885706 (+ 5735us) op_driver.cc:464] REPLICATION: finished
1214 20:52:21.885856 (+ 150us) write_op.cc:301] APPLY: starting
1214 20:52:21.885874 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:21.885917 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:21.885961 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:21.886022 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:21.886059 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:21.886065 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:21.886155 (+ 90us) log.cc:844] Serialized 23 byte log entry
1214 20:52:21.886232 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:21.886253 (+ 21us) write_op.cc:454] Released schema lock
1214 20:52:21.886261 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:21.886337 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:21.886340 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
1214 20:52:22.119874 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.120005 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.120010 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:22.120016 (+ 6us) tablet.cc:660] Decoding operations
1214 20:52:22.120111 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.120120 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.120125 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.120182 (+ 57us) tablet.cc:699] Row locks acquired
1214 20:52:22.120185 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:22.120218 (+ 33us) write_op.cc:270] Start()
1214 20:52:22.120267 (+ 49us) write_op.cc:276] Timestamp: P: 1765745542120197 usec, L: 0
1214 20:52:22.120271 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.120455 (+ 184us) log.cc:844] Serialized 356 byte log entry
1214 20:52:22.190093 (+ 69638us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.190383 (+ 290us) write_op.cc:301] APPLY: starting
1214 20:52:22.190400 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.190444 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.190477 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.190535 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.190570 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.190577 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.190659 (+ 82us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.190736 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.190757 (+ 21us) write_op.cc:454] Released schema lock
1214 20:52:22.190764 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.190836 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.190840 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1214 20:52:22.207867 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.207998 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.208005 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:22.208008 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:22.208110 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.208121 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.208126 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.208197 (+ 71us) tablet.cc:699] Row locks acquired
1214 20:52:22.208199 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:22.208238 (+ 39us) write_op.cc:270] Start()
1214 20:52:22.208273 (+ 35us) write_op.cc:276] Timestamp: P: 1765745542208199 usec, L: 0
1214 20:52:22.208277 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.275870 (+ 67593us) log.cc:844] Serialized 356 byte log entry
1214 20:52:22.276090 (+ 220us) spinlock_profiling.cc:243] Waited 67.4 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 000000000098492a 00007ff413306922 00007ff4146e9875 00007ff4146e8399 00007ff4146f07c0 00007ff4146f05cd 0000000000a8970c 00007ff40748e20f 00007ff40749152b 00007ff4074913a8 0000000000a8970c
1214 20:52:22.288575 (+ 12485us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.288645 (+ 70us) write_op.cc:301] APPLY: starting
1214 20:52:22.288657 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.288691 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.288694 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.288735 (+ 41us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.288763 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.288770 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.288830 (+ 60us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.288887 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.288901 (+ 14us) write_op.cc:454] Released schema lock
1214 20:52:22.288908 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":4455,"reported_tablets":152,"child_traces":[["op",{"apply.queue_time_us":96,"apply.run_cpu_time_us":474,"apply.run_wall_time_us":473,"num_ops":1,"prepare.queue_time_us":5030,"prepare.run_cpu_time_us":867,"prepare.run_wall_time_us":867,"raft.queue_time_us":2836,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":288,"replication_time_us":5924}],["op",{"apply.queue_time_us":237,"apply.run_cpu_time_us":445,"apply.run_wall_time_us":444,"num_ops":1,"prepare.queue_time_us":119624,"prepare.run_cpu_time_us":727,"prepare.run_wall_time_us":6383,"raft.queue_time_us":49,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":69788}],["op",{"apply.queue_time_us":30,"apply.run_cpu_time_us":325,"apply.run_wall_time_us":324,"num_ops":1,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":1622,"prepare.run_wall_time_us":68293,"raft.queue_time_us":1976,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":291,"replication_time_us":80273,"spinlock_wait_cycles":155026560}]]}
I20251214 20:52:23.117190 1202 ts_tablet_manager.cc:1434] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.047s user 0.000s sys 0.002s
I20251214 20:52:23.117729 1202 tablet_bootstrap.cc:492] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:23.120234 743 tablet_service.cc:1505] Processing CreateTablet for tablet 6b482043af42485eb4c73d6eb864a963 (DEFAULT_TABLE table=default.Table_b5f628ed069a4b19996c9b579d7678bc [id=a03c04d94997449bac7447ed634577d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.120721 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b482043af42485eb4c73d6eb864a963. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.124357 2164 consensus_queue.cc:1048] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:23.125753 2079 raft_consensus.cc:493] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.125885 2079 raft_consensus.cc:515] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.126446 2079 leader_election.cc:290] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.127339 1206 raft_consensus.cc:359] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.127527 1206 raft_consensus.cc:385] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.127580 1206 raft_consensus.cc:740] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.128832 1206 consensus_queue.cc:260] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.130203 1206 ts_tablet_manager.cc:1434] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20251214 20:52:23.130527 1206 tablet_bootstrap.cc:492] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:23.130714 2079 consensus_queue.cc:1048] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:23.131300 989 consensus_peers.cc:597] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:23.134352 2143 raft_consensus.cc:697] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:23.134765 2143 consensus_queue.cc:237] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.138281 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e31615d1277412eb8bd80451f2c4366" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:23.138491 909 raft_consensus.cc:2468] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:23.139163 991 leader_election.cc:304] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:23.139389 2170 raft_consensus.cc:2804] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:23.139500 2170 raft_consensus.cc:493] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:23.139595 2170 raft_consensus.cc:3060] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.139024 1650 consensus_queue.cc:1048] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 20:52:23.139858 1856 raft_consensus.cc:2243] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:23.140396 1856 raft_consensus.cc:2272] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.142661 2170 raft_consensus.cc:515] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.143807 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e31615d1277412eb8bd80451f2c4366" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:23.143976 904 raft_consensus.cc:3060] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.146710 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 41effce1ac504f27b619a4d9950202f9 (DEFAULT_TABLE table=default.Table_0fc25951e7cc4aebac50ad75316773f5 [id=d4196eff6f0c4965bfb9aa723570f6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.147316 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41effce1ac504f27b619a4d9950202f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.147490 744 tablet_service.cc:1505] Processing CreateTablet for tablet 41effce1ac504f27b619a4d9950202f9 (DEFAULT_TABLE table=default.Table_0fc25951e7cc4aebac50ad75316773f5 [id=d4196eff6f0c4965bfb9aa723570f6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.147873 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41effce1ac504f27b619a4d9950202f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.149538 1982 ts_tablet_manager.cc:1916] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.150074 904 raft_consensus.cc:2468] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:23.154892 991 leader_election.cc:304] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:23.155272 2165 raft_consensus.cc:2804] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:23.155498 2165 raft_consensus.cc:697] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:23.155880 1856 ts_tablet_manager.cc:1916] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.155860 2165 consensus_queue.cc:237] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.156059 1202 tablet_bootstrap.cc:654] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.159158 1977 ts_tablet_manager.cc:1929] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:23.159408 1977 log.cc:1199] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/3e5e220d083243eb8aa4328f366cd7ff
I20251214 20:52:23.159955 1977 ts_tablet_manager.cc:1950] T 3e5e220d083243eb8aa4328f366cd7ff P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:23.161453 2170 leader_election.cc:290] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.162447 768 raft_consensus.cc:3060] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.165071 776 raft_consensus.cc:3060] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.165452 1206 tablet_bootstrap.cc:654] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.166395 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e31615d1277412eb8bd80451f2c4366" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:23.166632 773 raft_consensus.cc:2468] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:23.178043 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_3b0a7d868e794df9bf7ec67f8b299c35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:23.178762 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b0a7d868e794df9bf7ec67f8b299c35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:23.183274 448 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:50727 (request call id 184) took 1096 ms (client timeout 1000 ms). Trace:
I20251214 20:52:23.190223 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e31615d1277412eb8bd80451f2c4366" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:23.190426 770 raft_consensus.cc:3060] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.190696 748 tablet_service.cc:1505] Processing CreateTablet for tablet 25180d5352204cdca3c49f907a587cb2 (DEFAULT_TABLE table=default.Table_ca7b705bd1754ce19fbd7cda96364054 [id=525aea874ba84a65af4b2cbbed3fbf9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.191212 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25180d5352204cdca3c49f907a587cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.194972 770 raft_consensus.cc:2468] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:23.196929 2211 tablet.cc:1722] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:23.204864 743 tablet_service.cc:1505] Processing CreateTablet for tablet 1def5f0de5ce4eb0bed36f626c4f0683 (DEFAULT_TABLE table=default.Table_0e2b5f246065408b923f196c464eceb7 [id=33a87231c7084c9b86bd0cc0cb9dc49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.205197 2168 consensus_queue.cc:1048] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:23.206941 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1def5f0de5ce4eb0bed36f626c4f0683. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:23.214270 448 rpcz_store.cc:269] 1214 20:52:22.086499 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.222337 (+135838us) service_pool.cc:225] Handling call
1214 20:52:22.278517 (+ 56180us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b17aa08 00007ff411c87ca6 00007ff411c879d5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:23.183224 (+904707us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.575747 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.575872 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.575877 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:22.575879 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:22.575984 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.575995 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.575999 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.576069 (+ 70us) tablet.cc:699] Row locks acquired
1214 20:52:22.576071 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:22.576105 (+ 34us) write_op.cc:270] Start()
1214 20:52:22.576131 (+ 26us) write_op.cc:276] Timestamp: P: 1765745542575994 usec, L: 0
1214 20:52:22.576135 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.576305 (+ 170us) log.cc:844] Serialized 527 byte log entry
1214 20:52:22.578829 (+ 2524us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.578973 (+ 144us) write_op.cc:301] APPLY: starting
1214 20:52:22.578985 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.579020 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.579048 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.579095 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.579161 (+ 66us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.579169 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:22.579237 (+ 68us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.579294 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.579307 (+ 13us) write_op.cc:454] Released schema lock
1214 20:52:22.579312 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:22.579374 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.579378 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1214 20:52:22.745366 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.745540 (+ 174us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.745548 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:22.745550 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:22.745702 (+ 152us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.745718 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.745722 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.745815 (+ 93us) tablet.cc:699] Row locks acquired
1214 20:52:22.745817 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:22.745864 (+ 47us) write_op.cc:270] Start()
1214 20:52:22.745909 (+ 45us) write_op.cc:276] Timestamp: P: 1765745542745801 usec, L: 0
1214 20:52:22.745912 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.746068 (+ 156us) log.cc:844] Serialized 356 byte log entry
1214 20:52:22.808582 (+ 62514us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.810022 (+ 1440us) write_op.cc:301] APPLY: starting
1214 20:52:22.810033 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.810056 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.810085 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.810123 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.810146 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.810153 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.810201 (+ 48us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.810243 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.810257 (+ 14us) write_op.cc:454] Released schema lock
1214 20:52:22.810261 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":88798,"reported_tablets":159,"spinlock_wait_cycles":39601408,"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":394,"apply.run_wall_time_us":402,"num_ops":1,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":647,"prepare.run_wall_time_us":647,"raft.queue_time_us":69,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":2670}],["op",{"apply.queue_time_us":1410,"apply.run_cpu_time_us":289,"apply.run_wall_time_us":290,"num_ops":1,"prepare.queue_time_us":36345,"prepare.run_cpu_time_us":770,"prepare.run_wall_time_us":1075,"replication_time_us":62655,"thread_start_us":5446,"threads_started":1}]]}
I20251214 20:52:23.215427 2217 tablet.cc:1722] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:23.215905 448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_488f7bbd18364d1e99b1f57aa7aba2c1" } modify_external_catalogs: true
I20251214 20:52:23.216110 448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_488f7bbd18364d1e99b1f57aa7aba2c1" } modify_external_catalogs: true
W20251214 20:52:23.192266 447 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:46855 (request call id 164) took 935 ms (client timeout 1000 ms). Trace:
I20251214 20:52:23.212285 452 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d9749ec3d4b8476a9e3d18ff2d300544" } new_table_name: "default.Table_912c9fc9392c4d99aa8485af7917eb85" modify_external_catalogs: true
I20251214 20:52:23.212973 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_08488285c9094a9dacc1e489c50271b3" } modify_external_catalogs: true
I20251214 20:52:23.218176 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_08488285c9094a9dacc1e489c50271b3" } modify_external_catalogs: true
W20251214 20:52:23.217993 447 rpcz_store.cc:269] 1214 20:52:22.257355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.294666 (+ 37311us) service_pool.cc:225] Handling call
1214 20:52:22.301486 (+ 6820us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b17aa08 00007ff411c87ca6 00007ff411c879d5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:23.192178 (+890692us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:22.639214 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:22.639340 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:22.639347 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:22.639349 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:22.639463 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:22.639478 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:22.639482 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:22.639549 (+ 67us) tablet.cc:699] Row locks acquired
1214 20:52:22.639551 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:22.639617 (+ 66us) write_op.cc:270] Start()
1214 20:52:22.639657 (+ 40us) write_op.cc:276] Timestamp: P: 1765745542639578 usec, L: 0
1214 20:52:22.639661 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:22.639884 (+ 223us) log.cc:844] Serialized 527 byte log entry
1214 20:52:22.642882 (+ 2998us) op_driver.cc:464] REPLICATION: finished
1214 20:52:22.644937 (+ 2055us) write_op.cc:301] APPLY: starting
1214 20:52:22.644954 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:22.644998 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:22.645031 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:22.645087 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:22.645119 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:22.645126 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:22.645195 (+ 69us) log.cc:844] Serialized 23 byte log entry
1214 20:52:22.645262 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:22.645281 (+ 19us) write_op.cc:454] Released schema lock
1214 20:52:22.645287 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":156939,"reported_tablets":151,"spinlock_wait_cycles":9519872,"child_traces":[["op",{"apply.queue_time_us":1982,"apply.run_cpu_time_us":418,"apply.run_wall_time_us":417,"num_ops":1,"prepare.queue_time_us":7731,"prepare.run_cpu_time_us":796,"prepare.run_wall_time_us":797,"raft.queue_time_us":34,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":268,"replication_time_us":3193}]]}
I20251214 20:52:23.218868 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_13f016135de44cf78187fbc4f9f8c622" } modify_external_catalogs: true
I20251214 20:52:23.219027 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_13f016135de44cf78187fbc4f9f8c622" } modify_external_catalogs: true
I20251214 20:52:23.223014 2167 raft_consensus.cc:493] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.223196 2167 raft_consensus.cc:515] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:23.223255 439 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 3a23561b9f4e48bbbd796e14fafec086 (table default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3 [id=f1338e81f4ec4dce87130f2ddbe98041], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3a23561b9f4e48bbbd796e14fafec086 consensus state has no leader
I20251214 20:52:23.224066 2167 leader_election.cc:290] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.226962 1856 ts_tablet_manager.cc:1929] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.227216 1856 log.cc:1199] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/2a7fc15882e2474bb6885070be13b1fa
I20251214 20:52:23.227691 1856 ts_tablet_manager.cc:1950] T 2a7fc15882e2474bb6885070be13b1fa P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:23.229933 1206 tablet_bootstrap.cc:492] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:23.230206 1206 ts_tablet_manager.cc:1403] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.100s user 0.005s sys 0.000s
I20251214 20:52:23.231055 1206 raft_consensus.cc:359] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.231397 1206 raft_consensus.cc:385] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.231489 1206 raft_consensus.cc:740] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.231798 1206 consensus_queue.cc:260] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.233177 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 146) took 2352 ms. Trace:
I20251214 20:52:23.233345 1856 rpcz_store.cc:276] 1214 20:52:20.880222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.880313 (+ 91us) service_pool.cc:225] Handling call
1214 20:52:23.066197 (+2185884us) ts_tablet_manager.cc:1151] Deleting tablet 2a7fc15882e2474bb6885070be13b1fa
1214 20:52:23.066205 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.140174 (+ 73969us) spinlock_profiling.cc:243] Waited 73.4 ms on lock 0x616000103954. stack: 00007fd8605acee6 00007fd8605acca0 00007fd85f5883ad 0000000000984f94 0000000000984f5d 000000000098492a 00007fd86c4961e0 00007fd86d7e40b3 00007fd8703a4757 00007fd8703d7744 00007fd8703b8240 00007fd8703b7fcd 0000000000a8970c 00007fd8605f320f 00007fd8605f652b 00007fd8605f63a8
1214 20:52:23.158744 (+ 18570us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.196133 (+ 37389us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.233157 (+ 37024us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":168849280,"tablet-delete.queue_time_us":2185728}
I20251214 20:52:23.233634 1206 ts_tablet_manager.cc:1434] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:23.233997 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 147) took 2352 ms. Trace:
I20251214 20:52:23.234156 1856 rpcz_store.cc:276] 1214 20:52:20.881251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.881398 (+ 147us) service_pool.cc:225] Handling call
1214 20:52:23.233890 (+2352492us) ts_tablet_manager.cc:1151] Deleting tablet 9523d72fd91445539d67e1eac2a80afd
1214 20:52:23.233901 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.233982 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352337}
I20251214 20:52:23.234479 1856 tablet_replica.cc:333] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:23.234831 1856 raft_consensus.cc:2243] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.235064 1856 raft_consensus.cc:2272] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.236086 1856 ts_tablet_manager.cc:1916] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.236569 1206 tablet_bootstrap.cc:492] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:23.238104 1769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:34819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=34369
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 14 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9516
ASAN enabled
I20251214 20:52:23.240247 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 1def5f0de5ce4eb0bed36f626c4f0683 (DEFAULT_TABLE table=default.Table_0e2b5f246065408b923f196c464eceb7 [id=33a87231c7084c9b86bd0cc0cb9dc49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.240423 1022 tablet_service.cc:1505] Processing CreateTablet for tablet cea9423c40044e4cabb8bbd755a46688 (DEFAULT_TABLE table=default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.241140 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 25180d5352204cdca3c49f907a587cb2 (DEFAULT_TABLE table=default.Table_ca7b705bd1754ce19fbd7cda96364054 [id=525aea874ba84a65af4b2cbbed3fbf9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.242309 2079 consensus_queue.cc:1048] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:23.245760 2211 tablet.cc:1722] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:23.257170 2228 tablet.cc:1722] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:23.261775 1769 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 20:52:23.259119 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea9423c40044e4cabb8bbd755a46688. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.267817 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 6b482043af42485eb4c73d6eb864a963 (DEFAULT_TABLE table=default.Table_b5f628ed069a4b19996c9b579d7678bc [id=a03c04d94997449bac7447ed634577d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.268260 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b482043af42485eb4c73d6eb864a963. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.268486 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1def5f0de5ce4eb0bed36f626c4f0683. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.271307 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25180d5352204cdca3c49f907a587cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.257937 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408013cf49e46d0b18d24b48483ea49" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:23.276510 989 leader_election.cc:343] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.279261 2221 tablet.cc:1722] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:23.279850 1982 ts_tablet_manager.cc:1929] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.279992 1982 log.cc:1199] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/a0279992d1f2419ab1caae116a0391b2
I20251214 20:52:23.280376 1982 ts_tablet_manager.cc:1950] T a0279992d1f2419ab1caae116a0391b2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:23.280714 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 113) took 3054 ms. Trace:
I20251214 20:52:23.282392 1856 ts_tablet_manager.cc:1929] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.282512 1856 log.cc:1199] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/7fcfcfe81fe24d369eea696efc61d21c
I20251214 20:52:23.260682 1203 tablet_bootstrap.cc:492] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:23.282707 1203 ts_tablet_manager.cc:1403] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.172s user 0.003s sys 0.000s
I20251214 20:52:23.282815 1856 ts_tablet_manager.cc:1950] T 7fcfcfe81fe24d369eea696efc61d21c P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:23.283093 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 148) took 2401 ms. Trace:
I20251214 20:52:23.283200 1856 rpcz_store.cc:276] 1214 20:52:20.881700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.881756 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:23.234437 (+2352681us) ts_tablet_manager.cc:1151] Deleting tablet 7fcfcfe81fe24d369eea696efc61d21c
1214 20:52:23.234445 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.259108 (+ 24663us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.282289 (+ 23181us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.283077 (+ 788us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352468}
I20251214 20:52:23.283526 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 149) took 2400 ms. Trace:
I20251214 20:52:23.283581 1203 raft_consensus.cc:359] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.283782 1203 raft_consensus.cc:385] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.283842 1203 raft_consensus.cc:740] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.284047 1203 consensus_queue.cc:260] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.284453 1206 tablet_bootstrap.cc:654] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.284551 1203 ts_tablet_manager.cc:1434] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:23.284859 1203 tablet_bootstrap.cc:492] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:23.280802 1982 rpcz_store.cc:276] 1214 20:52:20.226654 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.226704 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:23.082774 (+2856070us) ts_tablet_manager.cc:1151] Deleting tablet a0279992d1f2419ab1caae116a0391b2
1214 20:52:23.082788 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.206861 (+124073us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.279712 (+ 72851us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.280696 (+ 984us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39313,"tablet-delete.queue_time_us":2709048}
I20251214 20:52:23.286738 2168 raft_consensus.cc:493] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.283599 1856 rpcz_store.cc:276] 1214 20:52:20.882568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.882626 (+ 58us) service_pool.cc:225] Handling call
1214 20:52:23.283441 (+2400815us) ts_tablet_manager.cc:1151] Deleting tablet f0d661a1bb034053a4634ff537eafc7d
1214 20:52:23.283448 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.283513 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2400612}
I20251214 20:52:23.286900 2168 raft_consensus.cc:515] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.287055 1856 tablet_replica.cc:333] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:23.287513 2168 leader_election.cc:290] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.289811 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 114) took 3062 ms. Trace:
I20251214 20:52:23.292865 1769 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251214 20:52:23.260321 2222 tablet.cc:1722] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:23.261729 768 raft_consensus.cc:3060] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.258826 2227 tablet.cc:1722] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:23.304051 1982 rpcz_store.cc:276] 1214 20:52:20.226944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.226994 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:23.289646 (+3062652us) ts_tablet_manager.cc:1151] Deleting tablet bf8643c7e4094f529cd55656c7b3edb6
1214 20:52:23.289657 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.289788 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2915876}
I20251214 20:52:23.304780 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet a0279992d1f2419ab1caae116a0391b2 (table default.Table_4e46507057c94aa7a347491bf2e482ff [id=3906806ef8294342a79b9face8753212]) successfully deleted
I20251214 20:52:23.305424 1982 tablet_replica.cc:333] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.305747 1982 raft_consensus.cc:2243] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.306002 1982 raft_consensus.cc:2272] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:23.306783 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet bf8643c7e4094f529cd55656c7b3edb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf8643c7e4094f529cd55656c7b3edb6 already in progress: creating tablet
I20251214 20:52:23.307041 1982 ts_tablet_manager.cc:1916] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:23.307796 989 consensus_peers.cc:597] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:23.307335 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca57963bcfb54eec9fcbed82757d8387" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:23.305193 1206 tablet_bootstrap.cc:492] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:23.308329 1206 ts_tablet_manager.cc:1403] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
I20251214 20:52:23.309202 1206 raft_consensus.cc:359] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:23.309291 989 leader_election.cc:343] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.309480 1206 raft_consensus.cc:385] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.309712 1206 raft_consensus.cc:740] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.310382 1206 consensus_queue.cc:260] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.312050 1206 ts_tablet_manager.cc:1434] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:23.320230 1206 tablet_bootstrap.cc:492] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:23.327143 1856 raft_consensus.cc:2243] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:23.327553 1856 raft_consensus.cc:2272] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.327895 1206 tablet_bootstrap.cc:654] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.167632 1202 tablet_bootstrap.cc:492] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:23.171895 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 65e79db74fa046698507c83416ee1669 with delete_type TABLET_DATA_DELETED (replaced by tablet 4dcd511a5dca4a2cbda81fbc255ce354) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.328269 1202 ts_tablet_manager.cc:1403] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.211s user 0.003s sys 0.000s
I20251214 20:52:23.172243 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 242a77ae57174de48f4eaf43384ee0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.173079 885 tablet_service.cc:1505] Processing CreateTablet for tablet 54e5fff1fc84419ab3d3131cb3a12d92 (DEFAULT_TABLE table=default.Table_1e0e46b1f1e1418eadafc528585358f3 [id=d99b78f1c7dd40af85d84125b089518e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.173960 894 tablet_service.cc:1552] Processing DeleteTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f with delete_type TABLET_DATA_DELETED (replaced by tablet f11834cabdad445b8e53cd237d146282) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.328891 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e5fff1fc84419ab3d3131cb3a12d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.329182 1202 raft_consensus.cc:359] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.329375 1202 raft_consensus.cc:385] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.329466 1202 raft_consensus.cc:740] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.329685 1202 consensus_queue.cc:260] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.182396 2213 tablet.cc:1722] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:23.184015 880 tablet_service.cc:1505] Processing CreateTablet for tablet 336cb9d03fd44f70930662ee3c8d987a (DEFAULT_TABLE table=default.Table_ebccdf7648384d44b9629a1b41509e68 [id=2e1e99c74ef5467190886342facb1a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.330787 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 336cb9d03fd44f70930662ee3c8d987a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.184464 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 35a529702f924a5c9bf53c3c668c6094 with delete_type TABLET_DATA_DELETED (replaced by tablet 719d157ac8f74eb7911df05ab5e4a3d5) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.332242 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 023299180d074893847e84dcd9812751
I20251214 20:52:23.332873 453 catalog_manager.cc:6724] ReplaceTablet: tablet 023299180d074893847e84dcd9812751 deleted and replaced by tablet 67104e34df294ef5986171eca79cdc72
I20251214 20:52:23.333254 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a889ffcf345c4209b084b35515555d67" } modify_external_catalogs: true
I20251214 20:52:23.333508 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a889ffcf345c4209b084b35515555d67" } modify_external_catalogs: true
I20251214 20:52:23.334095 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_53e58d3cdbfe4ab2945a18979051d7a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:23.334785 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53e58d3cdbfe4ab2945a18979051d7a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.184738 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 69f6108792fa4c2a9744e3248c76251f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.336546 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 2a7fc15882e2474bb6885070be13b1fa (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]) successfully deleted
I20251214 20:52:23.336791 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 023299180d074893847e84dcd9812751 with delete_type TABLET_DATA_DELETED (replaced by tablet 67104e34df294ef5986171eca79cdc72) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:23.336869 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 9523d72fd91445539d67e1eac2a80afd because the tablet was not found. No further retry: Not found: Tablet not found: 9523d72fd91445539d67e1eac2a80afd
I20251214 20:52:23.337446 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 7fcfcfe81fe24d369eea696efc61d21c (table default.Table_caa5b0dabd9d4ea8a8b16c550407dc53 [id=4c909a51df124835bd806b8c6e53214e]) successfully deleted
W20251214 20:52:23.337774 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f0d661a1bb034053a4634ff537eafc7d because the tablet was not found. No further retry: Not found: Tablet not found: f0d661a1bb034053a4634ff537eafc7d
I20251214 20:52:23.185189 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d89ee602c524c96a21c0a235af07481 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e11723c042a4637839f1b13bed0935a) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.186345 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 1dca369dc5f54a4ea1783e334c5670d6) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.193526 879 tablet_service.cc:1505] Processing CreateTablet for tablet 41effce1ac504f27b619a4d9950202f9 (DEFAULT_TABLE table=default.Table_0fc25951e7cc4aebac50ad75316773f5 [id=d4196eff6f0c4965bfb9aa723570f6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.339685 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41effce1ac504f27b619a4d9950202f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.194101 882 tablet_service.cc:1505] Processing CreateTablet for tablet 6b482043af42485eb4c73d6eb864a963 (DEFAULT_TABLE table=default.Table_b5f628ed069a4b19996c9b579d7678bc [id=a03c04d94997449bac7447ed634577d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.194304 888 tablet_service.cc:1505] Processing CreateTablet for tablet 1def5f0de5ce4eb0bed36f626c4f0683 (DEFAULT_TABLE table=default.Table_0e2b5f246065408b923f196c464eceb7 [id=33a87231c7084c9b86bd0cc0cb9dc49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.343492 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b482043af42485eb4c73d6eb864a963. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.345678 2228 tablet.cc:1722] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:23.346473 751 tablet_service.cc:1505] Processing CreateTablet for tablet 0dbfe8ee68bf4a2c88489b1f81b36991 (DEFAULT_TABLE table=default.Table_00bef63ea36c4a7585c2832be1454b68 [id=6a279e57008549c593df838211cc9cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.346868 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dbfe8ee68bf4a2c88489b1f81b36991. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.194885 878 tablet_service.cc:1505] Processing CreateTablet for tablet cea9423c40044e4cabb8bbd755a46688 (DEFAULT_TABLE table=default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.195086 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dd657f3baff40bb8cb7f4648367f7a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.216544 2219 tablet.cc:1722] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:23.229166 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408013cf49e46d0b18d24b48483ea49" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:23.257699 2224 tablet.cc:1722] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:23.258258 2226 tablet.cc:1722] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:23.261653 2196 consensus_queue.cc:1048] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:23.290637 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca57963bcfb54eec9fcbed82757d8387" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:23.349359 915 raft_consensus.cc:2468] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
W20251214 20:52:23.349879 991 leader_election.cc:343] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.350013 991 leader_election.cc:304] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:23.350435 991 leader_election.cc:304] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:23.350715 2167 raft_consensus.cc:2749] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:23.350858 2167 raft_consensus.cc:2804] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:23.350905 2167 raft_consensus.cc:493] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:23.350953 2167 raft_consensus.cc:3060] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.353490 2167 raft_consensus.cc:515] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.355787 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 023299180d074893847e84dcd9812751 with delete_type TABLET_DATA_DELETED (replaced by tablet 67104e34df294ef5986171eca79cdc72) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.355962 884 tablet_service.cc:1505] Processing CreateTablet for tablet 25180d5352204cdca3c49f907a587cb2 (DEFAULT_TABLE table=default.Table_ca7b705bd1754ce19fbd7cda96364054 [id=525aea874ba84a65af4b2cbbed3fbf9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.356475 892 tablet_service.cc:1505] Processing CreateTablet for tablet 0dbfe8ee68bf4a2c88489b1f81b36991 (DEFAULT_TABLE table=default.Table_00bef63ea36c4a7585c2832be1454b68 [id=6a279e57008549c593df838211cc9cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.356616 1203 tablet_bootstrap.cc:654] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.356827 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 3e5e220d083243eb8aa4328f366cd7ff (table default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]) successfully deleted
I20251214 20:52:23.298175 1650 consensus_queue.cc:1048] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:23.313510 2198 raft_consensus.cc:493] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.359230 2143 consensus_queue.cc:1048] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:23.359366 2198 raft_consensus.cc:515] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.360148 2143 raft_consensus.cc:493] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63848a9fb78a4013a6f1657621272d75)
I20251214 20:52:23.360180 2198 leader_election.cc:290] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:23.360335 2143 raft_consensus.cc:515] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.361078 2143 leader_election.cc:290] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:23.361799 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25180d5352204cdca3c49f907a587cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.362254 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea9423c40044e4cabb8bbd755a46688. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.362427 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa1a44452074ec28a1bd9c48a5f47cb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:23.363319 848 leader_election.cc:343] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.364539 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa1a44452074ec28a1bd9c48a5f47cb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:23.365774 712 consensus_peers.cc:597] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bbb28259c1442ae94f4eff76c25c539. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:23.365810 851 leader_election.cc:343] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: c4b22043d81b493fb12116422b1d3fb1
W20251214 20:52:23.366106 851 leader_election.cc:343] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.366223 851 leader_election.cc:304] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:23.367462 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1def5f0de5ce4eb0bed36f626c4f0683. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.371343 1977 tablet_replica.cc:333] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:23.371672 1977 raft_consensus.cc:2243] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.371901 1977 raft_consensus.cc:2272] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.372325 2227 tablet.cc:1722] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251214 20:52:23.372666 848 leader_election.cc:343] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: c4b22043d81b493fb12116422b1d3fb1
I20251214 20:52:23.372817 848 leader_election.cc:304] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:23.372727 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 0cafed2861e94cd9b0d10d33864ba45d because the tablet was not found. No further retry: Not found: Tablet not found: 0cafed2861e94cd9b0d10d33864ba45d
I20251214 20:52:23.373209 2167 leader_election.cc:290] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.373422 1650 raft_consensus.cc:2749] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:23.373874 744 tablet_service.cc:1505] Processing CreateTablet for tablet cea9423c40044e4cabb8bbd755a46688 (DEFAULT_TABLE table=default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.374513 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 023299180d074893847e84dcd9812751 with delete_type TABLET_DATA_DELETED (replaced by tablet 67104e34df294ef5986171eca79cdc72) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.374634 1856 ts_tablet_manager.cc:1916] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.379693 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca57963bcfb54eec9fcbed82757d8387" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:23.381307 989 leader_election.cc:343] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.388018 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea9423c40044e4cabb8bbd755a46688. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.391919 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf
I20251214 20:52:23.392163 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet d01db63fd5d2474a9defdce0b98fc29d
I20251214 20:52:23.393520 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a79a6c9f5a07476a84b34893480c5601" } modify_external_catalogs: true
I20251214 20:52:23.393770 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a79a6c9f5a07476a84b34893480c5601" } modify_external_catalogs: true
W20251214 20:52:23.395392 452 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c2a6c6c0b6e46bcb7658fbeced6f738 consensus state has no leader
I20251214 20:52:23.395488 448 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet ae948ffc81764692b92045f5befe0184
I20251214 20:52:23.396333 452 master_service.cc:954] ReplaceTablet: received request to replace tablet fe24e4f4e74040249068505058c15622 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:23.396217 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ec9d2344b1bc430784fe937726619f77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.397097 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.397207 452 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3b13fb90bb2c4a0db268b7777b08f240" } new_table_name: "default.Table_5c7eb24742b84515af287c257637e51b" modify_external_catalogs: true
W20251214 20:52:23.397648 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec9d2344b1bc430784fe937726619f77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.398624 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet d01db63fd5d2474a9defdce0b98fc29d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.399586 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet ae948ffc81764692b92045f5befe0184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.400674 1203 tablet_bootstrap.cc:492] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:23.401917 1977 ts_tablet_manager.cc:1916] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.404649 2155 raft_consensus.cc:493] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63848a9fb78a4013a6f1657621272d75)
I20251214 20:52:23.404855 2155 raft_consensus.cc:515] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:23.405622 2155 leader_election.cc:290] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:23.409818 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.410076 743 tablet_service.cc:1552] Processing DeleteTablet for tablet d01db63fd5d2474a9defdce0b98fc29d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.410075 748 tablet_service.cc:1552] Processing DeleteTablet for tablet ae948ffc81764692b92045f5befe0184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:23.410749 713 leader_election.cc:343] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:23.411418 2227 tablet.cc:1722] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:23.412982 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_e552103bd07e4f68b25576d6044d8861"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:23.413957 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e552103bd07e4f68b25576d6044d8861 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.410442 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 0dbfe8ee68bf4a2c88489b1f81b36991 (DEFAULT_TABLE table=default.Table_00bef63ea36c4a7585c2832be1454b68 [id=6a279e57008549c593df838211cc9cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.416307 1203 ts_tablet_manager.cc:1403] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.132s user 0.005s sys 0.002s
I20251214 20:52:23.416754 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dbfe8ee68bf4a2c88489b1f81b36991. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.419250 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dbfe8ee68bf4a2c88489b1f81b36991. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.421316 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca57963bcfb54eec9fcbed82757d8387" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:23.421557 915 raft_consensus.cc:3060] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.423409 1206 tablet_bootstrap.cc:492] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:23.423722 1206 ts_tablet_manager.cc:1403] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.104s user 0.004s sys 0.001s
I20251214 20:52:23.424139 915 raft_consensus.cc:2468] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:23.424753 1206 raft_consensus.cc:359] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.425060 1206 raft_consensus.cc:385] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.425149 1206 raft_consensus.cc:740] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.425482 991 leader_election.cc:304] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:23.426215 2167 raft_consensus.cc:2804] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:23.426652 2167 raft_consensus.cc:697] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:23.427064 2167 consensus_queue.cc:237] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.427093 1206 consensus_queue.cc:260] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.427693 1203 raft_consensus.cc:359] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.427903 1203 raft_consensus.cc:385] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.427968 1203 raft_consensus.cc:740] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.428200 1203 consensus_queue.cc:260] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.428530 1982 ts_tablet_manager.cc:1929] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:23.428808 1203 ts_tablet_manager.cc:1434] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251214 20:52:23.429479 1203 tablet_bootstrap.cc:492] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:23.429536 1206 ts_tablet_manager.cc:1434] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251214 20:52:23.430662 1206 tablet_bootstrap.cc:492] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:23.431454 1203 tablet_bootstrap.cc:654] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.431602 1982 log.cc:1199] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/a34eb8fa3e34427495fa5985b7ba81bf
I20251214 20:52:23.439994 1206 tablet_bootstrap.cc:654] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.447005 1982 ts_tablet_manager.cc:1950] T a34eb8fa3e34427495fa5985b7ba81bf P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
W20251214 20:52:23.451085 713 consensus_peers.cc:597] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bbb28259c1442ae94f4eff76c25c539. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:23.451179 1977 ts_tablet_manager.cc:1929] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:23.451331 1977 log.cc:1199] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/8af99e0bd8774a678575847a9ec2e557
I20251214 20:52:23.451619 1977 ts_tablet_manager.cc:1950] T 8af99e0bd8774a678575847a9ec2e557 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:23.453083 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 115) took 3225 ms. Trace:
I20251214 20:52:23.453632 1982 rpcz_store.cc:276] 1214 20:52:20.227312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.227400 (+ 88us) service_pool.cc:225] Handling call
1214 20:52:23.305363 (+3077963us) ts_tablet_manager.cc:1151] Deleting tablet a34eb8fa3e34427495fa5985b7ba81bf
1214 20:52:23.305373 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.315584 (+ 10211us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.428407 (+112823us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.453003 (+ 24596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2931545}
I20251214 20:52:23.454663 1982 tablet_replica.cc:333] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.455518 1982 raft_consensus.cc:2243] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.456362 1982 raft_consensus.cc:2272] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.456442 1977 tablet_replica.cc:333] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:23.456727 1977 raft_consensus.cc:2243] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.456888 1977 raft_consensus.cc:2272] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.457429 1977 ts_tablet_manager.cc:1916] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.458582 1982 ts_tablet_manager.cc:1916] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.461160 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet a34eb8fa3e34427495fa5985b7ba81bf (table default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]) successfully deleted
I20251214 20:52:23.463021 892 tablet_service.cc:1552] Processing DeleteTablet for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.463165 1206 tablet_bootstrap.cc:492] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:23.463326 1206 ts_tablet_manager.cc:1403] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.000s
I20251214 20:52:23.464214 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 8af99e0bd8774a678575847a9ec2e557 (table default.Table_ade67d7e10e04c6fb0cfc134947a559d [id=7986a565ce914875bf3e24c1cc2c288a]) successfully deleted
I20251214 20:52:23.464215 1206 raft_consensus.cc:359] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.464543 1206 raft_consensus.cc:385] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251214 20:52:23.464532 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 16816fa6b8c34207bb929147910b93cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16816fa6b8c34207bb929147910b93cb already in progress: creating tablet
I20251214 20:52:23.464689 1206 raft_consensus.cc:740] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.465159 2143 raft_consensus.cc:2749] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:23.465472 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 99418.4us
I20251214 20:52:23.465526 1206 consensus_queue.cc:260] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.469134 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_e9b7a7c58ab24328adb161b3cdd1c356" } new_table_name: "default.Table_b87980e2da6a4052b109938fcf0527d8" modify_external_catalogs: true
I20251214 20:52:23.470466 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:23.470815 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:23.471417 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3d23e6dc5242452693b8aaac124a863f" } new_table_name: "default.Table_e3feb6d6cacb48dca8455eacf4cfb84b" modify_external_catalogs: true
I20251214 20:52:23.472241 447 catalog_manager.cc:5654] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:23.472728 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (table default.Table_89c0478b97144f35b1a0adeb64be2d9f [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]). Expected version 1 got 0
W20251214 20:52:23.473163 712 leader_election.cc:343] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:23.473342 712 leader_election.cc:304] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:23.473423 447 catalog_manager.cc:5654] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:23.473722 2155 raft_consensus.cc:2749] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:23.473781 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:23.477519 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_b72c21c5b00d430ea93eff8bf8bcb228"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.480399 1856 ts_tablet_manager.cc:1929] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251214 20:52:23.481034 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b72c21c5b00d430ea93eff8bf8bcb228 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.481333 1856 log.cc:1199] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/bb698ff30fcc47d09000cc68d2c5225b
I20251214 20:52:23.481950 1856 ts_tablet_manager.cc:1950] T bb698ff30fcc47d09000cc68d2c5225b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:23.483328 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 150) took 2599 ms. Trace:
I20251214 20:52:23.484615 1977 ts_tablet_manager.cc:1929] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:23.485598 1856 rpcz_store.cc:276] 1214 20:52:20.883643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.883715 (+ 72us) service_pool.cc:225] Handling call
1214 20:52:23.287004 (+2403289us) ts_tablet_manager.cc:1151] Deleting tablet bb698ff30fcc47d09000cc68d2c5225b
1214 20:52:23.287013 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.378654 (+ 91641us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.480281 (+101627us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.483305 (+ 3024us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2403066}
I20251214 20:52:23.486423 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet bb698ff30fcc47d09000cc68d2c5225b (table default.Table_7c95904dd8b044e4ad472681d5a5a274 [id=8c96acfc06ea4f00996fa066d719e418]) successfully deleted
I20251214 20:52:23.489158 1206 ts_tablet_manager.cc:1434] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20251214 20:52:23.499017 1206 tablet_bootstrap.cc:492] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:23.502123 1206 tablet_bootstrap.cc:654] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:23.502861 2240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:23.492425 888 tablet_service.cc:1552] Processing DeleteTablet for tablet d01db63fd5d2474a9defdce0b98fc29d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.492713 892 tablet_service.cc:1552] Processing DeleteTablet for tablet ae948ffc81764692b92045f5befe0184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.494519 1202 ts_tablet_manager.cc:1434] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.166s user 0.004s sys 0.012s
I20251214 20:52:23.512167 1202 tablet_bootstrap.cc:492] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:23.512344 1203 tablet_bootstrap.cc:492] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:23.512377 454 master_service.cc:954] ReplaceTablet: received request to replace tablet 2b92b6d5dde2465684d19aee44a83160 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:23.512476 1203 ts_tablet_manager.cc:1403] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.083s user 0.017s sys 0.005s
I20251214 20:52:23.513270 1203 raft_consensus.cc:359] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.513477 1203 raft_consensus.cc:385] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.513549 1203 raft_consensus.cc:740] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.513715 1202 tablet_bootstrap.cc:654] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.513850 1203 consensus_queue.cc:260] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.514397 1203 ts_tablet_manager.cc:1434] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:23.514596 1977 log.cc:1199] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/b315744c8cd043c5a8bdceec73a7e28b
I20251214 20:52:23.514565 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:23.514693 1203 tablet_bootstrap.cc:492] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:23.514914 1977 ts_tablet_manager.cc:1950] T b315744c8cd043c5a8bdceec73a7e28b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:23.516366 1977 tablet_replica.cc:333] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:23.516561 1977 raft_consensus.cc:2243] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:23.516884 1977 raft_consensus.cc:2272] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.517614 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 151) took 2633 ms. Trace:
W20251214 20:52:23.517524 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 75405.8us
I20251214 20:52:23.517741 1856 rpcz_store.cc:276] 1214 20:52:20.883786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.883965 (+ 179us) service_pool.cc:225] Handling call
1214 20:52:23.517490 (+2633525us) ts_tablet_manager.cc:1151] Deleting tablet f3a2143b5508445492b970d2588cf1cf
1214 20:52:23.517503 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.517594 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2633321}
I20251214 20:52:23.518158 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 152) took 2633 ms. Trace:
I20251214 20:52:23.518815 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet b315744c8cd043c5a8bdceec73a7e28b (table default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]) successfully deleted
I20251214 20:52:23.519006 1856 rpcz_store.cc:276] 1214 20:52:20.884853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.884916 (+ 63us) service_pool.cc:225] Handling call
1214 20:52:23.518072 (+2633156us) ts_tablet_manager.cc:1151] Deleting tablet 63497e77baeb4d328be2421b5732a28a
1214 20:52:23.518082 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.518141 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2632975}
W20251214 20:52:23.519204 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 58ac06f852fc4abea5cb213926002ca1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58ac06f852fc4abea5cb213926002ca1 already in progress: creating tablet
W20251214 20:52:23.519440 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet fe24e4f4e74040249068505058c15622 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24e4f4e74040249068505058c15622 already in progress: creating tablet
W20251214 20:52:23.519671 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 60ae3db968f6412aa0c3952fafe65f88 because the tablet was not found. No further retry: Not found: Tablet not found: 60ae3db968f6412aa0c3952fafe65f88
I20251214 20:52:23.520164 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 153) took 2634 ms. Trace:
W20251214 20:52:23.520252 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2da66ec83e9140f7a605f0c86abff468 because the tablet was not found. No further retry: Not found: Tablet not found: 2da66ec83e9140f7a605f0c86abff468
I20251214 20:52:23.520272 1856 rpcz_store.cc:276] 1214 20:52:20.885683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.885739 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:23.519658 (+2633919us) ts_tablet_manager.cc:1151] Deleting tablet 2da66ec83e9140f7a605f0c86abff468
1214 20:52:23.519667 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.520142 (+ 475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2633688}
I20251214 20:52:23.520615 1856 tablet_replica.cc:333] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:23.520838 1856 raft_consensus.cc:2243] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:23.520855 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f3a2143b5508445492b970d2588cf1cf because the tablet was not found. No further retry: Not found: Tablet not found: f3a2143b5508445492b970d2588cf1cf
I20251214 20:52:23.521042 1856 raft_consensus.cc:2272] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:23.521178 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 63497e77baeb4d328be2421b5732a28a because the tablet was not found. No further retry: Not found: Tablet not found: 63497e77baeb4d328be2421b5732a28a
W20251214 20:52:23.521440 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2da66ec83e9140f7a605f0c86abff468 because the tablet was not found. No further retry: Not found: Tablet not found: 2da66ec83e9140f7a605f0c86abff468
I20251214 20:52:23.521660 1856 ts_tablet_manager.cc:1916] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.522553 1977 ts_tablet_manager.cc:1916] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.526820 1203 tablet_bootstrap.cc:654] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.525739 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d6384d1f74bc4ec9a1985c5d8655e4b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:23.532601 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6384d1f74bc4ec9a1985c5d8655e4b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:23.537837 2243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:23.538372 1977 ts_tablet_manager.cc:1929] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.538686 1856 ts_tablet_manager.cc:1929] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.538837 1856 log.cc:1199] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/d6c0f445cd7e44769d79ad5566fd8d97
I20251214 20:52:23.539309 1856 ts_tablet_manager.cc:1950] T d6c0f445cd7e44769d79ad5566fd8d97 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:23.539454 1977 log.cc:1199] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/a063083f54ad49d2a842fa080d44ed85
I20251214 20:52:23.539855 1977 ts_tablet_manager.cc:1950] T a063083f54ad49d2a842fa080d44ed85 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:23.540642 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 154) took 2654 ms. Trace:
I20251214 20:52:23.541605 1856 rpcz_store.cc:276] 1214 20:52:20.886448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.886499 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:23.520568 (+2634069us) ts_tablet_manager.cc:1151] Deleting tablet d6c0f445cd7e44769d79ad5566fd8d97
1214 20:52:23.520577 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.529640 (+ 9063us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.538595 (+ 8955us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.540620 (+ 2025us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2633898}
I20251214 20:52:23.541919 1206 tablet_bootstrap.cc:492] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:23.542101 1206 ts_tablet_manager.cc:1403] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20251214 20:52:23.542560 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 155) took 2655 ms. Trace:
I20251214 20:52:23.542568 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet d6c0f445cd7e44769d79ad5566fd8d97 (table default.Table_d863e40d2a9e46649f9dc97f33a616db [id=cd65155def8244e59f08dac4d695567d]) successfully deleted
I20251214 20:52:23.542922 1977 tablet_replica.cc:333] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:23.542897 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 2b92b6d5dde2465684d19aee44a83160
I20251214 20:52:23.543228 1977 raft_consensus.cc:2243] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.543606 454 catalog_manager.cc:6724] ReplaceTablet: tablet 2b92b6d5dde2465684d19aee44a83160 deleted and replaced by tablet 6b389a63a58d438db41a343db6e04a88
I20251214 20:52:23.543638 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b92b6d5dde2465684d19aee44a83160 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b389a63a58d438db41a343db6e04a88) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.543383 1856 rpcz_store.cc:276] 1214 20:52:20.887169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.887220 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:23.542459 (+2655239us) ts_tablet_manager.cc:1151] Deleting tablet 472e4a5990e64d02b1d52dc6e86d7190
1214 20:52:23.542470 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.542546 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2655007}
I20251214 20:52:23.544047 1977 raft_consensus.cc:2272] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.544440 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 156) took 2656 ms. Trace:
W20251214 20:52:23.544800 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 472e4a5990e64d02b1d52dc6e86d7190 because the tablet was not found. No further retry: Not found: Tablet not found: 472e4a5990e64d02b1d52dc6e86d7190
I20251214 20:52:23.545245 1977 ts_tablet_manager.cc:1916] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.545171 1206 raft_consensus.cc:359] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:23.545348 1206 raft_consensus.cc:385] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.545405 1206 raft_consensus.cc:740] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.545608 1206 consensus_queue.cc:260] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:23.544936 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b92b6d5dde2465684d19aee44a83160 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b389a63a58d438db41a343db6e04a88) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.546069 1206 ts_tablet_manager.cc:1434] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:23.546097 1856 rpcz_store.cc:276] 1214 20:52:20.888166 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.888221 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:23.544355 (+2656134us) ts_tablet_manager.cc:1151] Deleting tablet b652a8a80246443491551e76e68d2a8b
1214 20:52:23.544364 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.544427 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2655936}
I20251214 20:52:23.546561 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 157) took 2656 ms. Trace:
W20251214 20:52:23.546643 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b652a8a80246443491551e76e68d2a8b because the tablet was not found. No further retry: Not found: Tablet not found: b652a8a80246443491551e76e68d2a8b
I20251214 20:52:23.546717 1856 rpcz_store.cc:276] 1214 20:52:20.890258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.890313 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:23.546486 (+2656173us) ts_tablet_manager.cc:1151] Deleting tablet c25832da5c1644329e18653000f3330e
1214 20:52:23.546494 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.546546 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2655966}
I20251214 20:52:23.547297 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 158) took 2655 ms. Trace:
W20251214 20:52:23.547487 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c25832da5c1644329e18653000f3330e because the tablet was not found. No further retry: Not found: Tablet not found: c25832da5c1644329e18653000f3330e
I20251214 20:52:23.547437 1856 rpcz_store.cc:276] 1214 20:52:20.891374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.891433 (+ 59us) service_pool.cc:225] Handling call
1214 20:52:23.547222 (+2655789us) ts_tablet_manager.cc:1151] Deleting tablet 6b4f582d8165439d86d9228c146dc41a
1214 20:52:23.547230 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.547281 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2655599}
W20251214 20:52:23.547972 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 6b4f582d8165439d86d9228c146dc41a because the tablet was not found. No further retry: Not found: Tablet not found: 6b4f582d8165439d86d9228c146dc41a
I20251214 20:52:23.551040 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet a063083f54ad49d2a842fa080d44ed85 (table default.Table_15002dcbc05542cebdf323d46303384b [id=71344b71ec00402f8df0c119fd8704bc]) successfully deleted
I20251214 20:52:23.551278 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b92b6d5dde2465684d19aee44a83160 with delete_type TABLET_DATA_DELETED (replaced by tablet 6b389a63a58d438db41a343db6e04a88) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:23.551365 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2d164747cb294bd4b126b4420231867d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d164747cb294bd4b126b4420231867d already in progress: creating tablet
I20251214 20:52:23.554482 449 master_service.cc:954] ReplaceTablet: received request to replace tablet 37a33cc9bb784f72956ac9f5493a0c3d from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:23.559975 1202 tablet_bootstrap.cc:492] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:23.560122 1202 ts_tablet_manager.cc:1403] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.001s
I20251214 20:52:23.560765 1202 raft_consensus.cc:359] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.560916 1202 raft_consensus.cc:385] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.560958 1202 raft_consensus.cc:740] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.561136 1202 consensus_queue.cc:260] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.561548 1202 ts_tablet_manager.cc:1434] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251214 20:52:23.561883 1202 tablet_bootstrap.cc:492] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:23.563222 748 tablet_service.cc:1505] Processing CreateTablet for tablet 880a78b924574181bdd804d599f4f579 (DEFAULT_TABLE table=default.Table_1f9ca4a533f64c5ebb5b98f0e25dfe76 [id=35891c12040241549d6723219c912da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.563601 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 880a78b924574181bdd804d599f4f579. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.564003 744 tablet_service.cc:1505] Processing CreateTablet for tablet 185049f1ef43499aa1d15d99549255f7 (DEFAULT_TABLE table=default.Table_3b0a7d868e794df9bf7ec67f8b299c35 [id=0e2f32942b8d49c68d9fd9212e0702aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.564289 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185049f1ef43499aa1d15d99549255f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.567353 743 tablet_service.cc:1505] Processing CreateTablet for tablet e68bf6b1a2974957adae70449334df64 (DEFAULT_TABLE table=default.Table_dce3448de74f4b92b7cc3de061c34dc7 [id=70039324533e4a70bb46118c2a0de77f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.567615 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68bf6b1a2974957adae70449334df64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.569440 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 880a78b924574181bdd804d599f4f579 (DEFAULT_TABLE table=default.Table_1f9ca4a533f64c5ebb5b98f0e25dfe76 [id=35891c12040241549d6723219c912da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.569782 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 880a78b924574181bdd804d599f4f579. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.569751 1206 tablet_bootstrap.cc:492] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:23.573077 751 tablet_service.cc:1505] Processing CreateTablet for tablet 47ca12fc0de44082aa3d4fabc919c1dc (DEFAULT_TABLE table=default.Table_02bb72c146234757b420555977074934 [id=5f186d0854254ce287edb0ccb77ee6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.573463 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ca12fc0de44082aa3d4fabc919c1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.575098 1024 tablet_service.cc:1505] Processing CreateTablet for tablet e68bf6b1a2974957adae70449334df64 (DEFAULT_TABLE table=default.Table_dce3448de74f4b92b7cc3de061c34dc7 [id=70039324533e4a70bb46118c2a0de77f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.575445 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68bf6b1a2974957adae70449334df64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.577273 1982 ts_tablet_manager.cc:1929] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.577400 1982 log.cc:1199] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/60ae3db968f6412aa0c3952fafe65f88
I20251214 20:52:23.577710 1982 ts_tablet_manager.cc:1950] T 60ae3db968f6412aa0c3952fafe65f88 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:23.578020 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 117) took 3349 ms. Trace:
I20251214 20:52:23.578097 1982 rpcz_store.cc:276] 1214 20:52:20.228066 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.368107 (+140041us) service_pool.cc:225] Handling call
1214 20:52:23.454561 (+3086454us) ts_tablet_manager.cc:1151] Deleting tablet 60ae3db968f6412aa0c3952fafe65f88
1214 20:52:23.454573 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.491778 (+ 37205us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.577145 (+ 85367us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.578003 (+ 858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3080491}
I20251214 20:52:23.578500 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 100) took 3362 ms. Trace:
I20251214 20:52:23.578557 1982 rpcz_store.cc:276] 1214 20:52:20.216165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.223979 (+ 7814us) service_pool.cc:225] Handling call
1214 20:52:23.578376 (+3354397us) ts_tablet_manager.cc:1151] Deleting tablet ec85857f047e44d78835305ddcb70768
1214 20:52:23.578384 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.578486 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3202824}
I20251214 20:52:23.578775 1982 tablet_replica.cc:333] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.578969 1982 raft_consensus.cc:2243] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.580102 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 60ae3db968f6412aa0c3952fafe65f88 (table default.Table_01dcfe83d7854b4e9b8d6f62a557a0fe [id=d180e3a95c0840589c0aeec13ef1a6ca]) successfully deleted
W20251214 20:52:23.580461 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ec85857f047e44d78835305ddcb70768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec85857f047e44d78835305ddcb70768 already in progress: creating tablet
I20251214 20:52:23.581103 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 37a33cc9bb784f72956ac9f5493a0c3d
I20251214 20:52:23.581583 449 catalog_manager.cc:6724] ReplaceTablet: tablet 37a33cc9bb784f72956ac9f5493a0c3d deleted and replaced by tablet b87b0871c2454971a15dd5643c155694
I20251214 20:52:23.581882 2231 consensus_queue.cc:1048] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:23.582720 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 37a33cc9bb784f72956ac9f5493a0c3d with delete_type TABLET_DATA_DELETED (replaced by tablet b87b0871c2454971a15dd5643c155694) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.583775 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_0f9951606583457cb248ab0e617b466b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.584942 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 37a33cc9bb784f72956ac9f5493a0c3d with delete_type TABLET_DATA_DELETED (replaced by tablet b87b0871c2454971a15dd5643c155694) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:23.586030 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f9951606583457cb248ab0e617b466b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.586551 1982 raft_consensus.cc:2272] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.586937 892 tablet_service.cc:1505] Processing CreateTablet for tablet 185049f1ef43499aa1d15d99549255f7 (DEFAULT_TABLE table=default.Table_3b0a7d868e794df9bf7ec67f8b299c35 [id=0e2f32942b8d49c68d9fd9212e0702aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.587360 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185049f1ef43499aa1d15d99549255f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.587813 1982 ts_tablet_manager.cc:1916] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.591661 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 3de3864300a94e53a856c6a93e8d6e80 (DEFAULT_TABLE table=default.Table_fc742d049909446bbdb1a0b0973734c0 [id=233d035cea634e6a82f2505aa9bf0e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.592032 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de3864300a94e53a856c6a93e8d6e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.592037 751 tablet_service.cc:1505] Processing CreateTablet for tablet 3de3864300a94e53a856c6a93e8d6e80 (DEFAULT_TABLE table=default.Table_fc742d049909446bbdb1a0b0973734c0 [id=233d035cea634e6a82f2505aa9bf0e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.592482 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de3864300a94e53a856c6a93e8d6e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.596097 1203 tablet_bootstrap.cc:492] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:23.596307 1203 ts_tablet_manager.cc:1403] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.082s user 0.001s sys 0.003s
I20251214 20:52:23.597154 1203 raft_consensus.cc:359] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.597420 1203 raft_consensus.cc:385] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.597517 1203 raft_consensus.cc:740] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.597788 1203 consensus_queue.cc:260] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.598351 1203 ts_tablet_manager.cc:1434] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:23.598675 1203 tablet_bootstrap.cc:492] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:23.598867 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_2be03ca8d3964e4db04c188da2e59bd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.599234 2155 raft_consensus.cc:493] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.599426 2155 raft_consensus.cc:515] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.600208 2155 leader_election.cc:290] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.600579 1203 tablet_bootstrap.cc:654] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.600914 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d89ee602c524c96a21c0a235af07481" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:23.601171 915 raft_consensus.cc:2468] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
W20251214 20:52:23.601552 713 leader_election.cc:343] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 9d89ee602c524c96a21c0a235af07481
I20251214 20:52:23.601965 1982 ts_tablet_manager.cc:1929] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.602085 1982 log.cc:1199] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:23.602497 712 leader_election.cc:304] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:23.602864 2250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:23.603053 2155 raft_consensus.cc:2804] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:23.603160 2155 raft_consensus.cc:493] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:23.603232 2155 raft_consensus.cc:3060] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.603506 1982 ts_tablet_manager.cc:1950] T 649f5bd67cbb4f61b7066269f0bf9266 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:23.603852 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 118) took 3235 ms. Trace:
I20251214 20:52:23.603922 1982 rpcz_store.cc:276] 1214 20:52:20.368458 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.368514 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:23.578721 (+3210207us) ts_tablet_manager.cc:1151] Deleting tablet 649f5bd67cbb4f61b7066269f0bf9266
1214 20:52:23.578729 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.594360 (+ 15631us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.601822 (+ 7462us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.603834 (+ 2012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3202766}
I20251214 20:52:23.604418 1982 tablet_replica.cc:333] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.604614 1982 raft_consensus.cc:2243] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.604794 1982 raft_consensus.cc:2272] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.605386 1982 ts_tablet_manager.cc:1916] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.605873 2155 raft_consensus.cc:515] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.607683 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d89ee602c524c96a21c0a235af07481" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:23.607914 915 raft_consensus.cc:3060] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:23.610177 713 leader_election.cc:343] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 9d89ee602c524c96a21c0a235af07481
I20251214 20:52:23.610587 915 raft_consensus.cc:2468] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:23.611413 712 leader_election.cc:304] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:23.611703 1963 raft_consensus.cc:2804] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:23.611811 1963 raft_consensus.cc:697] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:23.612028 1982 ts_tablet_manager.cc:1929] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.612115 1982 log.cc:1199] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/c16895ca08d94b56b5c4bc9352e330ef
I20251214 20:52:23.612066 1963 consensus_queue.cc:237] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.612478 1982 ts_tablet_manager.cc:1950] T c16895ca08d94b56b5c4bc9352e330ef P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:23.612776 888 tablet_service.cc:1505] Processing CreateTablet for tablet 880a78b924574181bdd804d599f4f579 (DEFAULT_TABLE table=default.Table_1f9ca4a533f64c5ebb5b98f0e25dfe76 [id=35891c12040241549d6723219c912da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.612958 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 119) took 3244 ms. Trace:
I20251214 20:52:23.613031 1982 rpcz_store.cc:276] 1214 20:52:20.368595 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.368717 (+ 122us) service_pool.cc:225] Handling call
1214 20:52:23.604365 (+3235648us) ts_tablet_manager.cc:1151] Deleting tablet c16895ca08d94b56b5c4bc9352e330ef
1214 20:52:23.604375 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.609085 (+ 4710us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.611923 (+ 2838us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.612940 (+ 1017us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3228337}
I20251214 20:52:23.613206 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 880a78b924574181bdd804d599f4f579. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.613338 1982 tablet_replica.cc:333] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.613531 1982 raft_consensus.cc:2243] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.613680 1982 raft_consensus.cc:2272] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.614257 1982 ts_tablet_manager.cc:1916] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.610596 1202 tablet_bootstrap.cc:654] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.617177 2155 leader_election.cc:290] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.620052 884 tablet_service.cc:1505] Processing CreateTablet for tablet e68bf6b1a2974957adae70449334df64 (DEFAULT_TABLE table=default.Table_dce3448de74f4b92b7cc3de061c34dc7 [id=70039324533e4a70bb46118c2a0de77f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.620420 1203 tablet_bootstrap.cc:492] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:23.620508 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68bf6b1a2974957adae70449334df64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.620577 1203 ts_tablet_manager.cc:1403] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20251214 20:52:23.621415 1203 raft_consensus.cc:359] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.621578 1203 raft_consensus.cc:385] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.621628 1203 raft_consensus.cc:740] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.621918 1203 consensus_queue.cc:260] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.622550 1203 ts_tablet_manager.cc:1434] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:23.622812 1203 tablet_bootstrap.cc:492] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:23.622889 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 185049f1ef43499aa1d15d99549255f7 (DEFAULT_TABLE table=default.Table_3b0a7d868e794df9bf7ec67f8b299c35 [id=0e2f32942b8d49c68d9fd9212e0702aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.623030 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 37a33cc9bb784f72956ac9f5493a0c3d with delete_type TABLET_DATA_DELETED (replaced by tablet b87b0871c2454971a15dd5643c155694) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.623329 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185049f1ef43499aa1d15d99549255f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.606737 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 649f5bd67cbb4f61b7066269f0bf9266 (table default.Table_cf3c60b13d0542f4ae62c69c088edafb [id=cc1f8c656d5a479caafc84000c78e7e1]) successfully deleted
I20251214 20:52:23.625489 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet c16895ca08d94b56b5c4bc9352e330ef (table default.Table_9f6bb794724346069627cd2f14a5cd1b [id=4e30619d28324239bc069a5ce053b167]) successfully deleted
W20251214 20:52:23.600050 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2be03ca8d3964e4db04c188da2e59bd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.626750 1203 tablet_bootstrap.cc:654] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.630278 1982 ts_tablet_manager.cc:1929] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:23.631325 1202 tablet_bootstrap.cc:492] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:23.631536 1202 ts_tablet_manager.cc:1403] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.003s
I20251214 20:52:23.631742 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:23.632606 1202 raft_consensus.cc:359] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.632786 1202 raft_consensus.cc:385] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.632836 1202 raft_consensus.cc:740] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.633122 892 tablet_service.cc:1505] Processing CreateTablet for tablet 47ca12fc0de44082aa3d4fabc919c1dc (DEFAULT_TABLE table=default.Table_02bb72c146234757b420555977074934 [id=5f186d0854254ce287edb0ccb77ee6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.633409 1982 log.cc:1199] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/25974e5db555447baf5799775d15cd14
I20251214 20:52:23.630396 1206 tablet_bootstrap.cc:654] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.633874 1982 ts_tablet_manager.cc:1950] T 25974e5db555447baf5799775d15cd14 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:23.634275 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 120) took 3265 ms. Trace:
I20251214 20:52:23.634393 1982 rpcz_store.cc:276] 1214 20:52:20.368738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.368850 (+ 112us) service_pool.cc:225] Handling call
1214 20:52:23.613290 (+3244440us) ts_tablet_manager.cc:1151] Deleting tablet 25974e5db555447baf5799775d15cd14
1214 20:52:23.613299 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.617170 (+ 3871us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.630142 (+ 12972us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.634256 (+ 4114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3237219}
I20251214 20:52:23.633204 1202 consensus_queue.cc:260] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.625042 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 47ca12fc0de44082aa3d4fabc919c1dc (DEFAULT_TABLE table=default.Table_02bb72c146234757b420555977074934 [id=5f186d0854254ce287edb0ccb77ee6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.635058 1202 ts_tablet_manager.cc:1434] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251214 20:52:23.635483 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 131) took 2941 ms. Trace:
I20251214 20:52:23.635630 1982 rpcz_store.cc:276] 1214 20:52:20.694097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.694188 (+ 91us) service_pool.cc:225] Handling call
1214 20:52:23.635339 (+2941151us) ts_tablet_manager.cc:1151] Deleting tablet 762068d560ff47918dd9fd8791083433
1214 20:52:23.635350 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.635465 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2940504}
I20251214 20:52:23.635718 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ca12fc0de44082aa3d4fabc919c1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.636059 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 132) took 2941 ms. Trace:
I20251214 20:52:23.636188 1982 rpcz_store.cc:276] 1214 20:52:20.694772 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.694883 (+ 111us) service_pool.cc:225] Handling call
1214 20:52:23.635959 (+2941076us) ts_tablet_manager.cc:1151] Deleting tablet 2accf8a160584c82bb695a52e3f2b05c
1214 20:52:23.635968 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.636043 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2930394}
I20251214 20:52:23.636564 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 133) took 2853 ms. Trace:
I20251214 20:52:23.636685 1982 rpcz_store.cc:276] 1214 20:52:20.783471 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.783538 (+ 67us) service_pool.cc:225] Handling call
1214 20:52:23.636467 (+2852929us) ts_tablet_manager.cc:1151] Deleting tablet 0d7c7b3f62764ff497f4d70b53eed7de
1214 20:52:23.636475 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.636547 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2852744}
I20251214 20:52:23.637063 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 134) took 2847 ms. Trace:
I20251214 20:52:23.637229 1982 rpcz_store.cc:276] 1214 20:52:20.789714 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.789792 (+ 78us) service_pool.cc:225] Handling call
1214 20:52:23.636968 (+2847176us) ts_tablet_manager.cc:1151] Deleting tablet 1dca369dc5f54a4ea1783e334c5670d6
1214 20:52:23.636976 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.637046 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2846960}
I20251214 20:52:23.637753 884 tablet_service.cc:1505] Processing CreateTablet for tablet 0c62ffb71fff43d4812eb86f40ce1c58 (DEFAULT_TABLE table=default.Table_ded13abcaf9545afad1de571fd382831 [id=3322c0d4eb1a409da134030cd0a993fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.638152 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 135) took 2802 ms. Trace:
I20251214 20:52:23.638269 1982 rpcz_store.cc:276] 1214 20:52:20.835291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.835360 (+ 69us) service_pool.cc:225] Handling call
1214 20:52:23.638060 (+2802700us) ts_tablet_manager.cc:1151] Deleting tablet 16816fa6b8c34207bb929147910b93cb
1214 20:52:23.638067 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.638137 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2802418}
I20251214 20:52:23.638427 1024 tablet_service.cc:1505] Processing CreateTablet for tablet e7b247b761c94c60ad3a468db371b30b (DEFAULT_TABLE table=default.Table_edb8b6390dba4303b887b41508f5f525 [id=e8a9bd4b63cf47babfa086577af02cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.638639 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 136) took 2791 ms. Trace:
I20251214 20:52:23.638701 1982 rpcz_store.cc:276] 1214 20:52:20.846926 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.847000 (+ 74us) service_pool.cc:225] Handling call
1214 20:52:23.638547 (+2791547us) ts_tablet_manager.cc:1151] Deleting tablet c8e33ba1149a44c294885d76f676160a
1214 20:52:23.638554 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.638625 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2785672}
I20251214 20:52:23.639057 1982 tablet_replica.cc:333] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.639325 1982 raft_consensus.cc:2243] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.639248 454 catalog_manager.cc:5654] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:23.639554 1982 raft_consensus.cc:2272] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.639628 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b247b761c94c60ad3a468db371b30b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.640260 1982 ts_tablet_manager.cc:1916] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.643589 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c62ffb71fff43d4812eb86f40ce1c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.644090 878 tablet_service.cc:1505] Processing CreateTablet for tablet e7b247b761c94c60ad3a468db371b30b (DEFAULT_TABLE table=default.Table_edb8b6390dba4303b887b41508f5f525 [id=e8a9bd4b63cf47babfa086577af02cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.644492 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b247b761c94c60ad3a468db371b30b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.635613 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 25974e5db555447baf5799775d15cd14 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]) successfully deleted
W20251214 20:52:23.645619 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 762068d560ff47918dd9fd8791083433 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 762068d560ff47918dd9fd8791083433 already in progress: creating tablet
W20251214 20:52:23.645877 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2accf8a160584c82bb695a52e3f2b05c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2accf8a160584c82bb695a52e3f2b05c already in progress: creating tablet
W20251214 20:52:23.646070 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 0d7c7b3f62764ff497f4d70b53eed7de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d7c7b3f62764ff497f4d70b53eed7de already in progress: creating tablet
W20251214 20:52:23.646260 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 1dca369dc5f54a4ea1783e334c5670d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dca369dc5f54a4ea1783e334c5670d6 already in progress: creating tablet
W20251214 20:52:23.646427 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 16816fa6b8c34207bb929147910b93cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16816fa6b8c34207bb929147910b93cb already in progress: creating tablet
W20251214 20:52:23.646595 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c8e33ba1149a44c294885d76f676160a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8e33ba1149a44c294885d76f676160a already in progress: creating tablet
I20251214 20:52:23.633718 449 master_service.cc:954] ReplaceTablet: received request to replace tablet fc7353593bda4fceaa389c393ddc19c6 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:23.648119 748 tablet_service.cc:1505] Processing CreateTablet for tablet e7b247b761c94c60ad3a468db371b30b (DEFAULT_TABLE table=default.Table_edb8b6390dba4303b887b41508f5f525 [id=e8a9bd4b63cf47babfa086577af02cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.648535 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b247b761c94c60ad3a468db371b30b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.649755 879 tablet_service.cc:1505] Processing CreateTablet for tablet 6d535d8f71b44c81bbba6937e804d900 (DEFAULT_TABLE table=default.Table_a91064ac3fb94282a02fbc867fdd581c [id=6b97974ca1764265aa6c291f8439da8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.650151 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d535d8f71b44c81bbba6937e804d900. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.650570 744 tablet_service.cc:1505] Processing CreateTablet for tablet 0c62ffb71fff43d4812eb86f40ce1c58 (DEFAULT_TABLE table=default.Table_ded13abcaf9545afad1de571fd382831 [id=3322c0d4eb1a409da134030cd0a993fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.650888 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c62ffb71fff43d4812eb86f40ce1c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.651363 743 tablet_service.cc:1505] Processing CreateTablet for tablet 6d535d8f71b44c81bbba6937e804d900 (DEFAULT_TABLE table=default.Table_a91064ac3fb94282a02fbc867fdd581c [id=6b97974ca1764265aa6c291f8439da8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.651661 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d535d8f71b44c81bbba6937e804d900. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.634254 888 tablet_service.cc:1505] Processing CreateTablet for tablet 3de3864300a94e53a856c6a93e8d6e80 (DEFAULT_TABLE table=default.Table_fc742d049909446bbdb1a0b0973734c0 [id=233d035cea634e6a82f2505aa9bf0e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.654237 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de3864300a94e53a856c6a93e8d6e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.657438 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 6d535d8f71b44c81bbba6937e804d900 (DEFAULT_TABLE table=default.Table_a91064ac3fb94282a02fbc867fdd581c [id=6b97974ca1764265aa6c291f8439da8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.657819 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d535d8f71b44c81bbba6937e804d900. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.658499 879 tablet_service.cc:1505] Processing CreateTablet for tablet c07966670d294a0495ec54d48c7f09cd (DEFAULT_TABLE table=default.Table_309db467eb00418598443e6f7a007e44 [id=8334b5c0fee9479e97df4162fade78a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.658547 751 tablet_service.cc:1505] Processing CreateTablet for tablet c07966670d294a0495ec54d48c7f09cd (DEFAULT_TABLE table=default.Table_309db467eb00418598443e6f7a007e44 [id=8334b5c0fee9479e97df4162fade78a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.658937 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c07966670d294a0495ec54d48c7f09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.659073 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c07966670d294a0495ec54d48c7f09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.664705 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 67104e34df294ef5986171eca79cdc72 (DEFAULT_TABLE table=default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.665004 1206 tablet_bootstrap.cc:492] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:23.665135 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67104e34df294ef5986171eca79cdc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.665164 1206 ts_tablet_manager.cc:1403] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.000s
I20251214 20:52:23.665949 1206 raft_consensus.cc:359] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.666144 1206 raft_consensus.cc:385] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.666196 1206 raft_consensus.cc:740] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.666435 1206 consensus_queue.cc:260] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.668506 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet fc7353593bda4fceaa389c393ddc19c6
I20251214 20:52:23.668738 1982 ts_tablet_manager.cc:1929] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.668856 1982 log.cc:1199] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/0cafed2861e94cd9b0d10d33864ba45d
I20251214 20:52:23.669075 449 catalog_manager.cc:6724] ReplaceTablet: tablet fc7353593bda4fceaa389c393ddc19c6 deleted and replaced by tablet a9c383e343544c83ada89bea6ab20e94
I20251214 20:52:23.669193 1982 ts_tablet_manager.cc:1950] T 0cafed2861e94cd9b0d10d33864ba45d P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:23.669484 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 137) took 2815 ms. Trace:
I20251214 20:52:23.669500 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_5a79c363f13a4849a6be94ea935e5b89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.669559 1982 rpcz_store.cc:276] 1214 20:52:20.854312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.855655 (+ 1343us) service_pool.cc:225] Handling call
1214 20:52:23.639013 (+2783358us) ts_tablet_manager.cc:1151] Deleting tablet 0cafed2861e94cd9b0d10d33864ba45d
1214 20:52:23.639021 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.643505 (+ 4484us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.667092 (+ 23587us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.669468 (+ 2376us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2783090}
I20251214 20:52:23.669961 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 154) took 1661 ms. Trace:
I20251214 20:52:23.670025 1982 rpcz_store.cc:276] 1214 20:52:22.008870 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.008957 (+ 87us) service_pool.cc:225] Handling call
1214 20:52:23.669835 (+1660878us) ts_tablet_manager.cc:1151] Deleting tablet 2d164747cb294bd4b126b4420231867d
1214 20:52:23.669844 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.669945 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660614}
W20251214 20:52:23.670122 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a79c363f13a4849a6be94ea935e5b89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.670248 1982 tablet_replica.cc:333] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.670439 1982 raft_consensus.cc:2243] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.670588 1982 raft_consensus.cc:2272] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.664705 1025 tablet_service.cc:1505] Processing CreateTablet for tablet c07966670d294a0495ec54d48c7f09cd (DEFAULT_TABLE table=default.Table_309db467eb00418598443e6f7a007e44 [id=8334b5c0fee9479e97df4162fade78a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.671422 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c07966670d294a0495ec54d48c7f09cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.671510 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 0cafed2861e94cd9b0d10d33864ba45d (table default.Table_ba612520b0be474db6573ce754044c8a [id=1bf28f656cfe496d8accd428d0b238f2]) successfully deleted
W20251214 20:52:23.671803 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2d164747cb294bd4b126b4420231867d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d164747cb294bd4b126b4420231867d already in progress: creating tablet
I20251214 20:52:23.673238 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet fc7353593bda4fceaa389c393ddc19c6 with delete_type TABLET_DATA_DELETED (replaced by tablet a9c383e343544c83ada89bea6ab20e94) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.673322 1206 ts_tablet_manager.cc:1434] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251214 20:52:23.633877 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 0c62ffb71fff43d4812eb86f40ce1c58 (DEFAULT_TABLE table=default.Table_ded13abcaf9545afad1de571fd382831 [id=3322c0d4eb1a409da134030cd0a993fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.673710 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c62ffb71fff43d4812eb86f40ce1c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.673902 879 tablet_service.cc:1552] Processing DeleteTablet for tablet fc7353593bda4fceaa389c393ddc19c6 with delete_type TABLET_DATA_DELETED (replaced by tablet a9c383e343544c83ada89bea6ab20e94) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.673645 1206 tablet_bootstrap.cc:492] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:23.674573 848 consensus_peers.cc:597] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:23.674762 1202 tablet_bootstrap.cc:492] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:23.675829 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ca12fc0de44082aa3d4fabc919c1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.676128 743 tablet_service.cc:1552] Processing DeleteTablet for tablet fc7353593bda4fceaa389c393ddc19c6 with delete_type TABLET_DATA_DELETED (replaced by tablet a9c383e343544c83ada89bea6ab20e94) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.676258 1202 tablet_bootstrap.cc:654] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.681576 1963 raft_consensus.cc:493] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.681751 1963 raft_consensus.cc:515] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.682430 1963 leader_election.cc:290] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.682687 1206 tablet_bootstrap.cc:654] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.682953 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a063083f54ad49d2a842fa080d44ed85" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:23.683157 1769 server_base.cc:1047] running on GCE node
W20251214 20:52:23.684216 712 leader_election.cc:343] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: a063083f54ad49d2a842fa080d44ed85
I20251214 20:52:23.684347 712 leader_election.cc:304] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:23.684368 743 tablet_service.cc:1505] Processing CreateTablet for tablet 67104e34df294ef5986171eca79cdc72 (DEFAULT_TABLE table=default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.684501 1982 ts_tablet_manager.cc:1916] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.684780 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67104e34df294ef5986171eca79cdc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.684986 1963 raft_consensus.cc:2749] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:23.687610 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 3a0f669313244e809c2868ca0ccfad65 (DEFAULT_TABLE table=default.Table_ee3a88061eb34c19b3fad30276bc7ece [id=0c56f14b40ac442994e232aa348d64a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.687970 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0f669313244e809c2868ca0ccfad65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.691880 1769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251214 20:52:23.691859 879 tablet_service.cc:1505] Processing CreateTablet for tablet 67104e34df294ef5986171eca79cdc72 (DEFAULT_TABLE table=default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.692360 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67104e34df294ef5986171eca79cdc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.697459 878 tablet_service.cc:1505] Processing CreateTablet for tablet 3a0f669313244e809c2868ca0ccfad65 (DEFAULT_TABLE table=default.Table_ee3a88061eb34c19b3fad30276bc7ece [id=0c56f14b40ac442994e232aa348d64a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.697919 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0f669313244e809c2868ca0ccfad65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.698464 748 tablet_service.cc:1505] Processing CreateTablet for tablet 7242be189ca54c9182181e33d6743e6b (DEFAULT_TABLE table=default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 [id=23efc0ee36244ca0a6ff0f2faeb00b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.699007 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7242be189ca54c9182181e33d6743e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.699020 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d46e98eafc574e9a92bd66399d44b0fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.704805 743 tablet_service.cc:1505] Processing CreateTablet for tablet 3a0f669313244e809c2868ca0ccfad65 (DEFAULT_TABLE table=default.Table_ee3a88061eb34c19b3fad30276bc7ece [id=0c56f14b40ac442994e232aa348d64a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.705228 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0f669313244e809c2868ca0ccfad65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.707322 878 tablet_service.cc:1505] Processing CreateTablet for tablet 7242be189ca54c9182181e33d6743e6b (DEFAULT_TABLE table=default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 [id=23efc0ee36244ca0a6ff0f2faeb00b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.707782 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7242be189ca54c9182181e33d6743e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.707821 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 7242be189ca54c9182181e33d6743e6b (DEFAULT_TABLE table=default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 [id=23efc0ee36244ca0a6ff0f2faeb00b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.708287 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7242be189ca54c9182181e33d6743e6b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:23.699884 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d46e98eafc574e9a92bd66399d44b0fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.727483 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 159) took 2835 ms. Trace:
I20251214 20:52:23.727707 1856 rpcz_store.cc:276] 1214 20:52:20.891528 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.891724 (+ 196us) service_pool.cc:225] Handling call
1214 20:52:23.727117 (+2835393us) ts_tablet_manager.cc:1151] Deleting tablet 2842f1dc0c374d24946f90dac1cf8aa5
1214 20:52:23.727167 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.727279 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2835193}
I20251214 20:52:23.728132 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 163) took 2731 ms. Trace:
I20251214 20:52:23.728241 1856 rpcz_store.cc:276] 1214 20:52:20.997195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:20.997254 (+ 59us) service_pool.cc:225] Handling call
1214 20:52:23.728015 (+2730761us) ts_tablet_manager.cc:1151] Deleting tablet 324379e3217442c3addcfe58060b9ad7
1214 20:52:23.728026 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.728101 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2730439}
I20251214 20:52:23.728540 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 165) took 2690 ms. Trace:
I20251214 20:52:23.728641 1856 rpcz_store.cc:276] 1214 20:52:21.037819 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.037870 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:23.728460 (+2690590us) ts_tablet_manager.cc:1151] Deleting tablet 6bd6856b800247de896f16eee66f2827
1214 20:52:23.728468 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.728512 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2690319}
I20251214 20:52:23.728917 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 166) took 2690 ms. Trace:
I20251214 20:52:23.729017 1856 rpcz_store.cc:276] 1214 20:52:21.038659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.038835 (+ 176us) service_pool.cc:225] Handling call
1214 20:52:23.728839 (+2690004us) ts_tablet_manager.cc:1151] Deleting tablet d768c45bc02b4d108798efbf3f6aaf7d
1214 20:52:23.728846 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.728890 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2689633}
I20251214 20:52:23.729346 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 167) took 2689 ms. Trace:
I20251214 20:52:23.729445 1856 rpcz_store.cc:276] 1214 20:52:21.039863 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.039915 (+ 52us) service_pool.cc:225] Handling call
1214 20:52:23.729244 (+2689329us) ts_tablet_manager.cc:1151] Deleting tablet 8d74d3abc45a41969289c0a0892bfa04
1214 20:52:23.729252 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.729319 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2689051}
I20251214 20:52:23.729713 1856 tablet_replica.cc:333] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:23.729974 1856 raft_consensus.cc:2243] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.730187 1856 raft_consensus.cc:2272] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.730885 1856 ts_tablet_manager.cc:1916] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:23.733119 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2842f1dc0c374d24946f90dac1cf8aa5 because the tablet was not found. No further retry: Not found: Tablet not found: 2842f1dc0c374d24946f90dac1cf8aa5
W20251214 20:52:23.733381 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 324379e3217442c3addcfe58060b9ad7 because the tablet was not found. No further retry: Not found: Tablet not found: 324379e3217442c3addcfe58060b9ad7
W20251214 20:52:23.733596 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 6bd6856b800247de896f16eee66f2827 because the tablet was not found. No further retry: Not found: Tablet not found: 6bd6856b800247de896f16eee66f2827
W20251214 20:52:23.733779 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet d768c45bc02b4d108798efbf3f6aaf7d because the tablet was not found. No further retry: Not found: Tablet not found: d768c45bc02b4d108798efbf3f6aaf7d
W20251214 20:52:23.734018 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8d74d3abc45a41969289c0a0892bfa04 because the tablet was not found. No further retry: Not found: Tablet not found: 8d74d3abc45a41969289c0a0892bfa04
I20251214 20:52:23.735294 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_1f9ca4a533f64c5ebb5b98f0e25dfe76" } modify_external_catalogs: true
I20251214 20:52:23.735524 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_1f9ca4a533f64c5ebb5b98f0e25dfe76" } modify_external_catalogs: true
I20251214 20:52:23.746567 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 880a78b924574181bdd804d599f4f579
I20251214 20:52:23.750790 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 880a78b924574181bdd804d599f4f579 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.751421 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 880a78b924574181bdd804d599f4f579 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.752420 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_dce3448de74f4b92b7cc3de061c34dc7" } modify_external_catalogs: true
I20251214 20:52:23.752645 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_dce3448de74f4b92b7cc3de061c34dc7" } modify_external_catalogs: true
I20251214 20:52:23.754832 1206 tablet_bootstrap.cc:492] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:23.755060 1206 ts_tablet_manager.cc:1403] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.008s
I20251214 20:52:23.755189 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 99663ff0fe094849acefe03db8d43b50 (DEFAULT_TABLE table=default.Table_581df19239f343f89fbe7e5c17d6d85c [id=2c6c7db9251948fe840e9fc0cc2076db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.755519 879 tablet_service.cc:1505] Processing CreateTablet for tablet 99663ff0fe094849acefe03db8d43b50 (DEFAULT_TABLE table=default.Table_581df19239f343f89fbe7e5c17d6d85c [id=2c6c7db9251948fe840e9fc0cc2076db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.755663 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99663ff0fe094849acefe03db8d43b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.756018 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99663ff0fe094849acefe03db8d43b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.759498 748 tablet_service.cc:1505] Processing CreateTablet for tablet 0a2fbec0543445b7856ef22c934c6840 (DEFAULT_TABLE table=default.Table_65d3039202a645e899e5f752e1e3db42 [id=b6bebe3ab7e2449dafc61fdda3e3c38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.759620 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 0a2fbec0543445b7856ef22c934c6840 (DEFAULT_TABLE table=default.Table_65d3039202a645e899e5f752e1e3db42 [id=b6bebe3ab7e2449dafc61fdda3e3c38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.759963 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2fbec0543445b7856ef22c934c6840. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.763163 1206 raft_consensus.cc:359] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.763350 1206 raft_consensus.cc:385] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.763398 1206 raft_consensus.cc:740] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.763617 1206 consensus_queue.cc:260] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.765259 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet e68bf6b1a2974957adae70449334df64
I20251214 20:52:23.765635 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2fbec0543445b7856ef22c934c6840. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.765920 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41]). Expected version 1 got 0
I20251214 20:52:23.767817 1206 ts_tablet_manager.cc:1434] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251214 20:52:23.768239 1206 tablet_bootstrap.cc:492] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:23.769763 879 tablet_service.cc:1552] Processing DeleteTablet for tablet e68bf6b1a2974957adae70449334df64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.766356 446 catalog_manager.cc:5654] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:23.769654 1206 tablet_bootstrap.cc:654] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.770861 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet a0976ffcc649461fac74e20b5e402275 (table default.Table_e9b7a7c58ab24328adb161b3cdd1c356 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 1 got 0
I20251214 20:52:23.771076 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 9da5e98c6f4843449886176e9fcfa8e1 (table default.Table_55f303d33ec84a269232fc1ba8f2d182 [id=5ded9fe825ae4308b5bd4b21c16634c8]). Expected version 1 got 0
I20251214 20:52:23.771471 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be]). Expected version 2 got 0
I20251214 20:52:23.771659 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738 (table default.Table_912c9fc9392c4d99aa8485af7917eb85 [id=a56279b96c2d45af90277c7636192936]). Expected version 1 got 0
I20251214 20:52:23.771951 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (table default.Table_89c0478b97144f35b1a0adeb64be2d9f [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]). Expected version 1 got 0
I20251214 20:52:23.772770 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:23.772924 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:23.773361 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet e6ebfc373fbf41b1920c271a15302484 (table default.Table_121c97f6b224464087b06c6fd31d4fa4 [id=1a3349421ffd4a7d933c8301fffb9154]). Expected version 1 got 0
I20251214 20:52:23.773496 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:23.773635 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
I20251214 20:52:23.773854 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 2f9def8e4fd449599820a63845397750 (table default.Table_99e702b2f3fc436db82c793284a1670c [id=a9ff75b0f47f4601b2d489c3e3d6fce0]). Expected version 1 got 0
I20251214 20:52:23.774049 751 tablet_service.cc:1552] Processing DeleteTablet for tablet e68bf6b1a2974957adae70449334df64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.774850 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 8c07b455b12145e28ecff35f2732b840 (DEFAULT_TABLE table=default.Table_dd6d020171b24861a33a63f13b128050 [id=aa336c07501b4a99afc12338d7556623]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.775115 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d528fb3647e34ed5b6b9199d7c6db3a5" } new_table_name: "default.Table_8f5893ce749b45e0889fb1735f5a59ba" modify_external_catalogs: true
I20251214 20:52:23.775192 879 tablet_service.cc:1505] Processing CreateTablet for tablet 8c07b455b12145e28ecff35f2732b840 (DEFAULT_TABLE table=default.Table_dd6d020171b24861a33a63f13b128050 [id=aa336c07501b4a99afc12338d7556623]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.775348 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c07b455b12145e28ecff35f2732b840. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.775553 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c07b455b12145e28ecff35f2732b840. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.774339 750 tablet_service.cc:1505] Processing CreateTablet for tablet 8c07b455b12145e28ecff35f2732b840 (DEFAULT_TABLE table=default.Table_dd6d020171b24861a33a63f13b128050 [id=aa336c07501b4a99afc12338d7556623]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.778182 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c07b455b12145e28ecff35f2732b840. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.778440 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 62ac29a23c174ad9a3d93a10af7fecbe (DEFAULT_TABLE table=default.Table_8ae98ffc7458410786e7844521f3c5c1 [id=4c6dacf190dd4c7aa5b4e056bf7dfd8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.779608 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62ac29a23c174ad9a3d93a10af7fecbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.787408 751 tablet_service.cc:1505] Processing CreateTablet for tablet 62ac29a23c174ad9a3d93a10af7fecbe (DEFAULT_TABLE table=default.Table_8ae98ffc7458410786e7844521f3c5c1 [id=4c6dacf190dd4c7aa5b4e056bf7dfd8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.787899 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62ac29a23c174ad9a3d93a10af7fecbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.777098 1982 ts_tablet_manager.cc:1929] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.789371 1982 log.cc:1199] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/e16fe751f10f4616a00d0af811998729
I20251214 20:52:23.789858 1982 ts_tablet_manager.cc:1950] T e16fe751f10f4616a00d0af811998729 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:23.793469 2165 raft_consensus.cc:493] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.793658 2165 raft_consensus.cc:515] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.794345 2165 leader_election.cc:290] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.795365 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa1a44452074ec28a1bd9c48a5f47cb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:23.795620 915 raft_consensus.cc:2468] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:23.796126 991 leader_election.cc:304] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:23.796572 2165 raft_consensus.cc:2804] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:23.796664 2165 raft_consensus.cc:493] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:23.796767 2165 raft_consensus.cc:3060] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.796814 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa1a44452074ec28a1bd9c48a5f47cb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:23.765231 743 tablet_service.cc:1505] Processing CreateTablet for tablet 99663ff0fe094849acefe03db8d43b50 (DEFAULT_TABLE table=default.Table_581df19239f343f89fbe7e5c17d6d85c [id=2c6c7db9251948fe840e9fc0cc2076db]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:23.797312 989 leader_election.cc:343] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.797634 1769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 20:52:23.797966 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99663ff0fe094849acefe03db8d43b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.798146 879 tablet_service.cc:1505] Processing CreateTablet for tablet 62ac29a23c174ad9a3d93a10af7fecbe (DEFAULT_TABLE table=default.Table_8ae98ffc7458410786e7844521f3c5c1 [id=4c6dacf190dd4c7aa5b4e056bf7dfd8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.798143 750 tablet_service.cc:1505] Processing CreateTablet for tablet 248e3666fd13434e97378eb5a98dfadd (DEFAULT_TABLE table=default.Table_3369f89218b140e7a52da8221726f4e1 [id=e901e2802c854b4f829e59e00aa90040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.799588 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 248e3666fd13434e97378eb5a98dfadd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.799814 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62ac29a23c174ad9a3d93a10af7fecbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.800815 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 880a78b924574181bdd804d599f4f579 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.803092 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 155) took 1794 ms. Trace:
I20251214 20:52:23.804494 1982 rpcz_store.cc:276] 1214 20:52:22.008984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.009268 (+ 284us) service_pool.cc:225] Handling call
1214 20:52:23.670208 (+1660940us) ts_tablet_manager.cc:1151] Deleting tablet e16fe751f10f4616a00d0af811998729
1214 20:52:23.670216 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.689287 (+ 19071us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.776999 (+ 87712us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.803073 (+ 26074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660748}
I20251214 20:52:23.806279 1769 hybrid_clock.cc:648] HybridClock initialized: now 1765745543806139 us; error 131 us; skew 500 ppm
I20251214 20:52:23.806630 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet e16fe751f10f4616a00d0af811998729 (table default.Table_7923a9cba4ce4b27a12617e394ffbf2e [id=8fd9be1206a74aa1921e0dade8be3774]) successfully deleted
I20251214 20:52:23.806887 1769 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 20:52:23.807159 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 153) took 1798 ms. Trace:
I20251214 20:52:23.807322 1982 rpcz_store.cc:276] 1214 20:52:22.008762 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.009311 (+ 549us) service_pool.cc:225] Handling call
1214 20:52:23.805263 (+1795952us) ts_tablet_manager.cc:1151] Deleting tablet 0c973e0111804b8b869068bdcc5869bf
1214 20:52:23.805273 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.806969 (+ 1696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1795681}
I20251214 20:52:23.807822 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 156) took 1798 ms. Trace:
W20251214 20:52:23.808117 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 0c973e0111804b8b869068bdcc5869bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c973e0111804b8b869068bdcc5869bf already in progress: creating tablet
I20251214 20:52:23.808393 1982 rpcz_store.cc:276] 1214 20:52:22.009128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.009497 (+ 369us) service_pool.cc:225] Handling call
1214 20:52:23.807706 (+1798209us) ts_tablet_manager.cc:1151] Deleting tablet 17550eb0ee6b476f8a8c304451f678b2
1214 20:52:23.807715 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.807804 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1798025}
I20251214 20:52:23.810001 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:23.810333 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 157) took 1801 ms. Trace:
I20251214 20:52:23.810418 1202 tablet_bootstrap.cc:492] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:23.810561 1202 ts_tablet_manager.cc:1403] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.136s user 0.018s sys 0.010s
W20251214 20:52:23.810639 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 17550eb0ee6b476f8a8c304451f678b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17550eb0ee6b476f8a8c304451f678b2 already in progress: creating tablet
I20251214 20:52:23.811112 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]). Expected version 1 got 0
I20251214 20:52:23.811367 1202 raft_consensus.cc:359] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:23.811532 1202 raft_consensus.cc:385] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.811580 1202 raft_consensus.cc:740] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.811803 1202 consensus_queue.cc:260] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:23.814000 1982 rpcz_store.cc:276] 1214 20:52:22.009246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.009706 (+ 460us) service_pool.cc:225] Handling call
1214 20:52:23.810238 (+1800532us) ts_tablet_manager.cc:1151] Deleting tablet 8eb1006b0ba440dcba2f9de04dca2702
1214 20:52:23.810246 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.810317 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800383}
W20251214 20:52:23.814872 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8eb1006b0ba440dcba2f9de04dca2702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8eb1006b0ba440dcba2f9de04dca2702 already in progress: creating tablet
I20251214 20:52:23.815668 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:23.814343 1982 tablet_replica.cc:333] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.815995 1982 raft_consensus.cc:2243] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.816036 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet e68bf6b1a2974957adae70449334df64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.816219 1982 raft_consensus.cc:2272] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.816483 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_1de37433cd074bd494bcaa2064ea1823 [id=611e3a3e47ba4f91a705e9b8d3c323f3]). Expected version 1 got 0
I20251214 20:52:23.816730 1202 ts_tablet_manager.cc:1434] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251214 20:52:23.817060 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:23.817528 1982 ts_tablet_manager.cc:1916] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.820992 1202 tablet_bootstrap.cc:492] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:23.821681 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d2688515bd1441fbad7516f8343ca775 (table default.Table_d2beb8dc1b054cada5e4c2ec9fabc8d4 [id=03997e2eab154ef18b2e3774b5984764]). Expected version 1 got 0
I20251214 20:52:23.816718 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 248e3666fd13434e97378eb5a98dfadd (DEFAULT_TABLE table=default.Table_3369f89218b140e7a52da8221726f4e1 [id=e901e2802c854b4f829e59e00aa90040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.822512 1202 tablet_bootstrap.cc:654] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.822206 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 8e11723c042a4637839f1b13bed0935a (table default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]). Expected version 1 got 0
I20251214 20:52:23.823006 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 248e3666fd13434e97378eb5a98dfadd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.826150 1206 tablet_bootstrap.cc:492] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
W20251214 20:52:23.826192 713 consensus_peers.cc:597] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6c0f445cd7e44769d79ad5566fd8d97. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:23.826298 1206 ts_tablet_manager.cc:1403] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.002s
I20251214 20:52:23.826565 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:23.827015 1206 raft_consensus.cc:359] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.827219 1206 raft_consensus.cc:385] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.827255 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet feb19ea389af4ea6bb158a0e8aaeb094 (table default.Table_7660878c81e04c17b9b6fa8d61ef43a6 [id=59d9716c5cee42c391018111323c5181]). Expected version 1 got 0
I20251214 20:52:23.827289 1206 raft_consensus.cc:740] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.827473 1206 consensus_queue.cc:260] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.828091 1206 ts_tablet_manager.cc:1434] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:23.828347 1206 tablet_bootstrap.cc:492] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:23.830197 1206 tablet_bootstrap.cc:654] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.831696 1202 tablet_bootstrap.cc:492] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:23.831835 1202 ts_tablet_manager.cc:1403] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20251214 20:52:23.827724 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 3a23561b9f4e48bbbd796e14fafec086 (table default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3 [id=f1338e81f4ec4dce87130f2ddbe98041]). Expected version 1 got 0
I20251214 20:52:23.832455 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d171301941fa43908e05984c3b016007 (table default.Table_5000227c87004ed7a604c72840d805da [id=7dbd8a94ed0b497d8eba92091819fc07]). Expected version 2 got 0
I20251214 20:52:23.832568 1202 raft_consensus.cc:359] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.832732 1202 raft_consensus.cc:385] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.832790 1202 raft_consensus.cc:740] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.832629 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 727e7b61ae3942098712720d344c5310 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]). Expected version 1 got 0
I20251214 20:52:23.833096 1202 consensus_queue.cc:260] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.835250 1202 ts_tablet_manager.cc:1434] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:23.835582 1202 tablet_bootstrap.cc:492] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:23.833550 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 80098b7c24824d129b8f25daf2115460 (table default.Table_4e0d19a09c1f44eb85c4a776f6cd599a [id=2a055be9028342be97f5c98f03f5a137]). Expected version 1 got 0
I20251214 20:52:23.836241 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_41d7d906f2714c57832f29b1a2692574 [id=9553b46856c94003a293135c814a7df7]). Expected version 1 got 0
I20251214 20:52:23.836678 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:23.837111 1202 tablet_bootstrap.cc:654] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.838280 1982 ts_tablet_manager.cc:1929] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:23.837139 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]). Expected version 1 got 0
I20251214 20:52:23.838688 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 566c6e4761d5486d964e276d151e32b0 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 2 got 0
I20251214 20:52:23.839104 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:23.839833 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 825baba561ee4a8ebd44f75d8c51706a
I20251214 20:52:23.840591 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_0dcbd217695546df9d1ca32583980907" } modify_external_catalogs: true
I20251214 20:52:23.840799 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_0dcbd217695546df9d1ca32583980907" } modify_external_catalogs: true
I20251214 20:52:23.842212 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 825baba561ee4a8ebd44f75d8c51706a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.842837 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 825baba561ee4a8ebd44f75d8c51706a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.843179 1982 log.cc:1199] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/3c3962c33a294fa98f8bd485268aee71
W20251214 20:52:23.843379 452 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db0ad1aa4193467db74cd707e7084b71 consensus state has no leader
I20251214 20:52:23.843799 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_9f0b2a3e0ef1496db7886462be5513c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.844250 454 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_5879a88313cb488b8092323f94dfd3a7 [id=bdb017130eed4d76b635603560c91d41] alter complete (version 1)
W20251214 20:52:23.844513 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f0b2a3e0ef1496db7886462be5513c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.845080 892 tablet_service.cc:1505] Processing CreateTablet for tablet 248e3666fd13434e97378eb5a98dfadd (DEFAULT_TABLE table=default.Table_3369f89218b140e7a52da8221726f4e1 [id=e901e2802c854b4f829e59e00aa90040]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.845587 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 248e3666fd13434e97378eb5a98dfadd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:23.845618 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 082ec70b8ee646b898d9201ad073d292 consensus state has no leader
I20251214 20:52:23.846007 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_bde8bb1617d3450f999e9a7960d96641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.846446 1769 webserver.cc:492] Webserver started at http://127.0.104.61:34369/ using document root <none> and password file <none>
W20251214 20:52:23.846493 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bde8bb1617d3450f999e9a7960d96641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:23.847589 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 088c72a6f5014b6fbc6ee060b6b7a590 consensus state has no leader
I20251214 20:52:23.847723 455 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 199) took 1020 ms. Trace:
I20251214 20:52:23.848053 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 3bd5c18696f54ea190c1ef84b93c4f63
I20251214 20:52:23.848731 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_32323ff67ecd4fe38fa02169904b88bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:23.849201 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32323ff67ecd4fe38fa02169904b88bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:23.850739 448 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0976ffcc649461fac74e20b5e402275 consensus state has no leader
I20251214 20:52:23.847812 455 rpcz_store.cc:276] 1214 20:52:22.827594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.084084 (+256490us) service_pool.cc:225] Handling call
1214 20:52:23.093865 (+ 9781us) spinlock_profiling.cc:243] Waited 9.47 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b182456 00007ff411c8e146 00007ff411c8de75 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:23.103886 (+ 10021us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:23.104120 (+ 234us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:23.104312 (+ 192us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:23.104432 (+ 120us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:23.376141 (+271709us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:23.847696 (+471555us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:23.140852 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:23.141158 (+ 306us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:23.141165 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:23.141168 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:23.141306 (+ 138us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:23.141316 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:23.141321 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:23.141381 (+ 60us) tablet.cc:699] Row locks acquired
1214 20:52:23.141383 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:23.141417 (+ 34us) write_op.cc:270] Start()
1214 20:52:23.141447 (+ 30us) write_op.cc:276] Timestamp: P: 1765745543141397 usec, L: 0
1214 20:52:23.141450 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:23.141938 (+ 488us) log.cc:844] Serialized 405 byte log entry
1214 20:52:23.329614 (+187676us) op_driver.cc:464] REPLICATION: finished
1214 20:52:23.332049 (+ 2435us) write_op.cc:301] APPLY: starting
1214 20:52:23.332058 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:23.332087 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:23.332111 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:23.332142 (+ 31us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:23.332164 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:23.332170 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:23.332218 (+ 48us) log.cc:844] Serialized 23 byte log entry
1214 20:52:23.332260 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:23.332271 (+ 11us) write_op.cc:454] Released schema lock
1214 20:52:23.332276 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:23.332332 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:23.332335 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":21781248,"child_traces":[["op",{"apply.queue_time_us":2410,"apply.run_cpu_time_us":276,"apply.run_wall_time_us":276,"num_ops":1,"prepare.queue_time_us":35929,"prepare.run_cpu_time_us":728,"prepare.run_wall_time_us":1538,"raft.queue_time_us":39,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":188150}]]}
I20251214 20:52:23.851366 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_5d1ff53eee3c4eb596c1b728e7d3bc1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.851118 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_da414f75240a40e6b20b644312217eae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:23.851852 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d1ff53eee3c4eb596c1b728e7d3bc1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:23.852177 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da414f75240a40e6b20b644312217eae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.854355 447 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_a258f399b3e44e009cf7e014db9079aa [id=ed335fb680bc441184e1682200ee68be] alter complete (version 2)
I20251214 20:52:23.854831 1982 ts_tablet_manager.cc:1950] T 3c3962c33a294fa98f8bd485268aee71 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
W20251214 20:52:23.855858 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9488c3505f5443998531cf749722be9 consensus state has no leader
I20251214 20:52:23.856141 449 master_service.cc:954] ReplaceTablet: received request to replace tablet 34df3a34b15b4fe19bd12f785b07a033 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:23.856503 1769 fs_manager.cc:362] Metadata directory not provided
I20251214 20:52:23.857228 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 158) took 1708 ms. Trace:
I20251214 20:52:23.857326 1982 rpcz_store.cc:276] 1214 20:52:22.148842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.149056 (+ 214us) service_pool.cc:225] Handling call
1214 20:52:23.814301 (+1665245us) ts_tablet_manager.cc:1151] Deleting tablet 3c3962c33a294fa98f8bd485268aee71
1214 20:52:23.814309 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.829770 (+ 15461us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.838150 (+ 8380us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.857196 (+ 19046us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627945}
I20251214 20:52:23.857633 1982 tablet_replica.cc:333] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.857815 1982 raft_consensus.cc:2243] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:23.858011 1769 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 20:52:23.858114 1982 pending_rounds.cc:70] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:23.858181 1982 pending_rounds.cc:77] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493741048455168 op_type: NO_OP noop_request { }
I20251214 20:52:23.858289 1982 raft_consensus.cc:2889] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:23.858379 1982 raft_consensus.cc:2272] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.858878 1982 ts_tablet_manager.cc:1916] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.827018 2165 raft_consensus.cc:515] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.860275 2165 leader_election.cc:290] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.860450 2165 raft_consensus.cc:493] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.860605 2165 raft_consensus.cc:515] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.861130 2165 leader_election.cc:290] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.836732 2184 raft_consensus.cc:493] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.861310 2184 raft_consensus.cc:515] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.861459 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_f4c54cf75bea445185ff40f822983fb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.861791 2184 leader_election.cc:290] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:23.862088 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4c54cf75bea445185ff40f822983fb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.863322 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa1a44452074ec28a1bd9c48a5f47cb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:23.863577 915 raft_consensus.cc:3060] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.858937 454 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9] alter complete (version 1)
I20251214 20:52:23.866590 915 raft_consensus.cc:2468] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:23.866768 1982 ts_tablet_manager.cc:1929] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.866874 1982 log.cc:1199] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/3bbb28259c1442ae94f4eff76c25c539
I20251214 20:52:23.867043 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebb26a5ae7224e9ca8a513c7b5013618" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:23.867491 1982 ts_tablet_manager.cc:1950] T 3bbb28259c1442ae94f4eff76c25c539 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:23.868263 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 159) took 1700 ms. Trace:
I20251214 20:52:23.868602 1982 rpcz_store.cc:276] 1214 20:52:22.167758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.167962 (+ 204us) service_pool.cc:225] Handling call
1214 20:52:23.857588 (+1689626us) ts_tablet_manager.cc:1151] Deleting tablet 3bbb28259c1442ae94f4eff76c25c539
1214 20:52:23.857597 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.863523 (+ 5926us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.866681 (+ 3158us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.868246 (+ 1565us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2,"tablet-delete.queue_time_us":1670421}
I20251214 20:52:23.869053 1982 tablet_replica.cc:333] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.869299 1982 raft_consensus.cc:2243] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.869509 1982 raft_consensus.cc:2272] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.869736 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa1a44452074ec28a1bd9c48a5f47cb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:23.869884 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebb26a5ae7224e9ca8a513c7b5013618" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:23.870045 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bd5c18696f54ea190c1ef84b93c4f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.870225 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2accf8a160584c82bb695a52e3f2b05c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:23.870683 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 3c3962c33a294fa98f8bd485268aee71 (table default.Table_a5388ce81c77421aa5c30bfe834f60d3 [id=93377015e22043a799548b2ac01c8826]) successfully deleted
I20251214 20:52:23.870815 1982 ts_tablet_manager.cc:1916] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.870936 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 3bbb28259c1442ae94f4eff76c25c539 (table default.Table_ab99f8e2fad843e6966b1d86e41d58f6 [id=b5bbf8858d18464a881a7c45090d397a]) successfully deleted
I20251214 20:52:23.871481 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_0c936131512c4d99b3999d74fabf18d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.872174 991 leader_election.cc:304] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:23.872521 991 leader_election.cc:343] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.872862 2167 raft_consensus.cc:2804] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:23.873021 2167 raft_consensus.cc:697] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:23.873308 2167 consensus_queue.cc:237] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:23.874576 989 leader_election.cc:343] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:23.875782 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c936131512c4d99b3999d74fabf18d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.875922 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2accf8a160584c82bb695a52e3f2b05c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:23.876183 915 raft_consensus.cc:2468] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:23.876616 991 leader_election.cc:304] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:23.876922 2184 raft_consensus.cc:2804] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:23.877038 2184 raft_consensus.cc:493] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:23.877130 2184 raft_consensus.cc:3060] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.877619 1982 ts_tablet_manager.cc:1929] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.877727 1982 log.cc:1199] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/c9b7149a436a4cab80b39adccf8567f9
W20251214 20:52:23.879422 989 leader_election.cc:343] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.879591 989 leader_election.cc:304] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:23.879531 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_9196cd658f9041269835197751563391"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.879768 455 master_service.cc:954] ReplaceTablet: received request to replace tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 from {username='slave'} at 127.0.0.1:50088
W20251214 20:52:23.880193 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9196cd658f9041269835197751563391 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:23.880193 989 leader_election.cc:343] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.880400 743 tablet_service.cc:1552] Processing DeleteTablet for tablet e16fe751f10f4616a00d0af811998729 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.880743 2165 raft_consensus.cc:2749] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:23.882026 2184 raft_consensus.cc:515] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.882045 1982 ts_tablet_manager.cc:1950] T c9b7149a436a4cab80b39adccf8567f9 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:23.882930 2184 leader_election.cc:290] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.883574 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2accf8a160584c82bb695a52e3f2b05c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:23.883634 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2accf8a160584c82bb695a52e3f2b05c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:23.883852 915 raft_consensus.cc:3060] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.883883 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 160) took 1716 ms. Trace:
I20251214 20:52:23.884084 1982 rpcz_store.cc:276] 1214 20:52:22.167904 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.168080 (+ 176us) service_pool.cc:225] Handling call
1214 20:52:23.869011 (+1700931us) ts_tablet_manager.cc:1151] Deleting tablet c9b7149a436a4cab80b39adccf8567f9
1214 20:52:23.869020 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.874140 (+ 5120us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.877528 (+ 3388us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.883338 (+ 5810us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1681778}
I20251214 20:52:23.884600 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 161) took 1710 ms. Trace:
I20251214 20:52:23.884747 1982 rpcz_store.cc:276] 1214 20:52:22.174243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.174302 (+ 59us) service_pool.cc:225] Handling call
1214 20:52:23.884514 (+1710212us) ts_tablet_manager.cc:1151] Deleting tablet 2842f1dc0c374d24946f90dac1cf8aa5
1214 20:52:23.884524 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.884585 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1697120}
I20251214 20:52:23.885082 1982 tablet_replica.cc:333] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.885318 1982 raft_consensus.cc:2243] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:23.885689 1982 raft_consensus.cc:2272] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.886292 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet c9b7149a436a4cab80b39adccf8567f9 (table default.Table_67ec8d0cd3f948218b4637de6d68a57b [id=bdc2d86a977d4146a1a86717c5916807]) successfully deleted
I20251214 20:52:23.886478 1982 ts_tablet_manager.cc:1916] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:23.886618 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2842f1dc0c374d24946f90dac1cf8aa5 because the tablet was not found. No further retry: Not found: Tablet not found: 2842f1dc0c374d24946f90dac1cf8aa5
I20251214 20:52:23.886725 915 raft_consensus.cc:2468] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:23.887223 991 leader_election.cc:304] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:23.887620 989 leader_election.cc:343] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.888051 2184 raft_consensus.cc:2804] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:23.888208 2184 raft_consensus.cc:697] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:23.888514 2184 consensus_queue.cc:237] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.892274 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fcfcfe81fe24d369eea696efc61d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.896934 452 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:50064 (request call id 4) took 2187 ms. Trace:
I20251214 20:52:23.897193 452 rpcz_store.cc:276] 1214 20:52:21.709310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.896602 (+2187292us) service_pool.cc:225] Handling call
1214 20:52:23.896888 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 20:52:23.903321 1020 tablet_service.cc:1552] Processing DeleteTablet for tablet bb698ff30fcc47d09000cc68d2c5225b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.903431 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:34819: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111),
I20251214 20:52:23.903622 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251214 20:52:23.843753 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 825baba561ee4a8ebd44f75d8c51706a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.903712 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d8bb40c2324c4c0eb7e52aa96e920813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.855278 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bd5c18696f54ea190c1ef84b93c4f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:23.904275 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8bb40c2324c4c0eb7e52aa96e920813 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.905282 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 762068d560ff47918dd9fd8791083433 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.910356 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet b69bb4d5979f492683fc13238a668b9f
I20251214 20:52:23.911602 888 tablet_service.cc:1505] Processing CreateTablet for tablet 0a2fbec0543445b7856ef22c934c6840 (DEFAULT_TABLE table=default.Table_65d3039202a645e899e5f752e1e3db42 [id=b6bebe3ab7e2449dafc61fdda3e3c38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.912014 878 tablet_service.cc:1552] Processing DeleteTablet for tablet b69bb4d5979f492683fc13238a668b9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.912325 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2fbec0543445b7856ef22c934c6840. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.912575 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet b69bb4d5979f492683fc13238a668b9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.915524 743 tablet_service.cc:1552] Processing DeleteTablet for tablet b69bb4d5979f492683fc13238a668b9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.916106 1203 tablet_bootstrap.cc:492] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:23.916275 1203 ts_tablet_manager.cc:1403] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.294s user 0.016s sys 0.007s
I20251214 20:52:23.917062 451 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (table default.Table_89c0478b97144f35b1a0adeb64be2d9f [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]). Expected version 1 got 0
I20251214 20:52:23.917086 1203 raft_consensus.cc:359] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.917268 1203 raft_consensus.cc:385] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.917317 1203 raft_consensus.cc:740] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.917512 1203 consensus_queue.cc:260] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.917927 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet fe0c3c8a1d5b46cfacd35d3f0d82e432
I20251214 20:52:23.918166 1203 ts_tablet_manager.cc:1434] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:23.918180 451 catalog_manager.cc:5654] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:23.918483 1203 tablet_bootstrap.cc:492] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:23.918841 451 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:23.918919 455 catalog_manager.cc:6724] ReplaceTablet: tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 deleted and replaced by tablet 15b3216f3d2b443e8ab2f994445956f9
I20251214 20:52:23.916455 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bd5c18696f54ea190c1ef84b93c4f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.920492 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 34df3a34b15b4fe19bd12f785b07a033
I20251214 20:52:23.920848 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b3216f3d2b443e8ab2f994445956f9) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.921546 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 34df3a34b15b4fe19bd12f785b07a033 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ec7824f5eff4cc59c45acd3b3b3d194) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.921964 449 catalog_manager.cc:6724] ReplaceTablet: tablet 34df3a34b15b4fe19bd12f785b07a033 deleted and replaced by tablet 5ec7824f5eff4cc59c45acd3b3b3d194
I20251214 20:52:23.922577 743 tablet_service.cc:1552] Processing DeleteTablet for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b3216f3d2b443e8ab2f994445956f9) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.922653 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 34df3a34b15b4fe19bd12f785b07a033 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ec7824f5eff4cc59c45acd3b3b3d194) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:23.919334 1977 ts_tablet_manager.cc:1929] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:23.922776 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 34df3a34b15b4fe19bd12f785b07a033 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ec7824f5eff4cc59c45acd3b3b3d194) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:23.922851 1977 log.cc:1199] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ca263a80702f4f64aa7c872c2be08207
I20251214 20:52:23.923348 879 tablet_service.cc:1552] Processing DeleteTablet for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b3216f3d2b443e8ab2f994445956f9) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:23.923601 1977 ts_tablet_manager.cc:1950] T ca263a80702f4f64aa7c872c2be08207 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:23.928392 1203 tablet_bootstrap.cc:654] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.934752 2231 raft_consensus.cc:493] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.935084 2231 raft_consensus.cc:515] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.936142 2231 leader_election.cc:290] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:23.936705 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408013cf49e46d0b18d24b48483ea49" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:23.936928 1051 raft_consensus.cc:2468] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:23.936686 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_47baf32513e6487fbd4ffde9bcba97ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:23.937425 851 leader_election.cc:304] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:23.937570 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47baf32513e6487fbd4ffde9bcba97ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:23.937732 2231 raft_consensus.cc:2804] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:23.937845 2231 raft_consensus.cc:493] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:23.937932 2231 raft_consensus.cc:3060] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.938433 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408013cf49e46d0b18d24b48483ea49" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:23.938943 848 leader_election.cc:343] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.940836 2231 raft_consensus.cc:515] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.941918 2231 leader_election.cc:290] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:23.941898 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408013cf49e46d0b18d24b48483ea49" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:23.942081 1051 raft_consensus.cc:3060] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.942153 2184 raft_consensus.cc:493] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:23.942294 2184 raft_consensus.cc:515] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.943297 2184 leader_election.cc:290] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:23.943565 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16816fa6b8c34207bb929147910b93cb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:23.943761 1206 tablet_bootstrap.cc:492] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:23.943929 1206 ts_tablet_manager.cc:1403] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.002s
W20251214 20:52:23.944101 991 leader_election.cc:343] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.944401 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408013cf49e46d0b18d24b48483ea49" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:23.944939 1206 raft_consensus.cc:359] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:23.945021 848 leader_election.cc:343] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.945154 1206 raft_consensus.cc:385] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.945238 1206 raft_consensus.cc:740] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.945477 1206 consensus_queue.cc:260] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.946965 1051 raft_consensus.cc:2468] T 4408013cf49e46d0b18d24b48483ea49 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:23.947793 1977 tablet_replica.cc:333] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:23.948055 1977 raft_consensus.cc:2243] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.948252 1977 raft_consensus.cc:2272] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.948663 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ca263a80702f4f64aa7c872c2be08207 (table default.Table_642da83f7ed0444688afdf91b7a9e7e8 [id=bca3a2d37f8b4fc797a5c52a43c14869]) successfully deleted
I20251214 20:52:23.948913 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16816fa6b8c34207bb929147910b93cb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:23.949005 851 leader_election.cc:304] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
W20251214 20:52:23.949738 989 leader_election.cc:343] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:23.949860 989 leader_election.cc:304] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:23.949939 1977 ts_tablet_manager.cc:1916] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.950520 2184 raft_consensus.cc:2749] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:23.953802 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_725af8309e494ad3a5a40a73d2f585fa" } modify_external_catalogs: true
I20251214 20:52:23.953999 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_725af8309e494ad3a5a40a73d2f585fa" } modify_external_catalogs: true
I20251214 20:52:23.955265 2231 raft_consensus.cc:2804] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:23.955771 1202 tablet_bootstrap.cc:492] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:23.956005 1202 ts_tablet_manager.cc:1403] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.120s user 0.005s sys 0.000s
I20251214 20:52:23.956501 2231 raft_consensus.cc:697] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:23.957093 2231 consensus_queue.cc:237] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.957469 1982 ts_tablet_manager.cc:1929] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:23.957455 1202 raft_consensus.cc:359] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.957600 1202 raft_consensus.cc:385] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.957715 1202 raft_consensus.cc:740] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.957983 1202 consensus_queue.cc:260] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.958127 1650 consensus_queue.cc:1048] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:23.958595 1202 ts_tablet_manager.cc:1434] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:23.958889 1202 tablet_bootstrap.cc:492] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:23.957598 1982 log.cc:1199] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/9c0f49e5d78240a296e47eccb3218e4f
I20251214 20:52:23.962841 1982 ts_tablet_manager.cc:1950] T 9c0f49e5d78240a296e47eccb3218e4f P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:23.963224 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 162) took 1786 ms. Trace:
I20251214 20:52:23.963266 1206 ts_tablet_manager.cc:1434] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20251214 20:52:23.963323 1982 rpcz_store.cc:276] 1214 20:52:22.176752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.176815 (+ 63us) service_pool.cc:225] Handling call
1214 20:52:23.885034 (+1708219us) ts_tablet_manager.cc:1151] Deleting tablet 9c0f49e5d78240a296e47eccb3218e4f
1214 20:52:23.885042 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.929009 (+ 43967us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.957361 (+ 28352us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:23.963206 (+ 5845us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1697604}
I20251214 20:52:23.963657 1982 tablet_replica.cc:333] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:23.963850 1982 raft_consensus.cc:2243] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:23.963955 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 9c0f49e5d78240a296e47eccb3218e4f (table default.Table_ec65ee1eda49435895dab50f78b4ee15 [id=8a0182244a254993998d4ea00d03cd9d]) successfully deleted
I20251214 20:52:23.964006 1982 raft_consensus.cc:2272] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:23.964498 1982 ts_tablet_manager.cc:1916] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:23.969991 449 catalog_manager.cc:5654] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:23.973500 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 0634838779854905b64dcdd028deb1e1 (DEFAULT_TABLE table=default.Table_5a79c363f13a4849a6be94ea935e5b89 [id=4093d9a107134e4b90970ddecc931769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.974061 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0634838779854905b64dcdd028deb1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.974591 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 47f610e6d08046a385c58274abab0e0b (DEFAULT_TABLE table=default.Table_2be03ca8d3964e4db04c188da2e59bd3 [id=1eeba8e8c0be4ac2b904a59363a89e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.975198 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f610e6d08046a385c58274abab0e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.975448 1206 tablet_bootstrap.cc:492] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:23.975762 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_989a947ef29f422cbcc7fd84b5cac678" } modify_external_catalogs: true
I20251214 20:52:23.976194 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_989a947ef29f422cbcc7fd84b5cac678" } modify_external_catalogs: true
I20251214 20:52:23.978106 1206 tablet_bootstrap.cc:654] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:23.978621 751 tablet_service.cc:1505] Processing CreateTablet for tablet 0634838779854905b64dcdd028deb1e1 (DEFAULT_TABLE table=default.Table_5a79c363f13a4849a6be94ea935e5b89 [id=4093d9a107134e4b90970ddecc931769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.979005 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0634838779854905b64dcdd028deb1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.981853 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 1c5ee4114cb34540af2c5b332ebe82ab (DEFAULT_TABLE table=default.Table_0f9951606583457cb248ab0e617b466b [id=3a0ead3ea5aa4cdea9e6d8fc156bee96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.982314 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5ee4114cb34540af2c5b332ebe82ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.983790 743 tablet_service.cc:1505] Processing CreateTablet for tablet 47f610e6d08046a385c58274abab0e0b (DEFAULT_TABLE table=default.Table_2be03ca8d3964e4db04c188da2e59bd3 [id=1eeba8e8c0be4ac2b904a59363a89e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.984079 1203 tablet_bootstrap.cc:492] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:23.984155 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f610e6d08046a385c58274abab0e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.984244 1203 ts_tablet_manager.cc:1403] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.000s
I20251214 20:52:23.985565 1203 raft_consensus.cc:359] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.985767 1203 raft_consensus.cc:385] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:23.985800 775 raft_consensus.cc:3060] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.985842 1203 raft_consensus.cc:740] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:23.986145 1203 consensus_queue.cc:260] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:23.987195 1203 ts_tablet_manager.cc:1434] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251214 20:52:23.987656 1032 tablet_service.cc:1505] Processing CreateTablet for tablet d5e0b89d02c04625b8e31910d60d5d40 (DEFAULT_TABLE table=default.Table_b72c21c5b00d430ea93eff8bf8bcb228 [id=4d54f97a2cde4143a185eb4af35bb07d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.987686 748 tablet_service.cc:1505] Processing CreateTablet for tablet 1c5ee4114cb34540af2c5b332ebe82ab (DEFAULT_TABLE table=default.Table_0f9951606583457cb248ab0e617b466b [id=3a0ead3ea5aa4cdea9e6d8fc156bee96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.988024 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5ee4114cb34540af2c5b332ebe82ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.988063 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e0b89d02c04625b8e31910d60d5d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.991652 750 tablet_service.cc:1505] Processing CreateTablet for tablet d5e0b89d02c04625b8e31910d60d5d40 (DEFAULT_TABLE table=default.Table_b72c21c5b00d430ea93eff8bf8bcb228 [id=4d54f97a2cde4143a185eb4af35bb07d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.992013 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e0b89d02c04625b8e31910d60d5d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.993273 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 2f97e5bad46343098418e8e34334b892 (DEFAULT_TABLE table=default.Table_ec9d2344b1bc430784fe937726619f77 [id=eedcb23f130b4d9b8789d27f24ccf9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.993649 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f97e5bad46343098418e8e34334b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.995558 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 639cf3b7654a4ae896c6edd8b2ff1844 (DEFAULT_TABLE table=default.Table_53e58d3cdbfe4ab2945a18979051d7a2 [id=02c00037bc944c00b0124d72af62e19b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.996093 738 tablet_service.cc:1505] Processing CreateTablet for tablet 2f97e5bad46343098418e8e34334b892 (DEFAULT_TABLE table=default.Table_ec9d2344b1bc430784fe937726619f77 [id=eedcb23f130b4d9b8789d27f24ccf9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.996218 744 tablet_service.cc:1505] Processing CreateTablet for tablet 639cf3b7654a4ae896c6edd8b2ff1844 (DEFAULT_TABLE table=default.Table_53e58d3cdbfe4ab2945a18979051d7a2 [id=02c00037bc944c00b0124d72af62e19b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:23.996490 768 raft_consensus.cc:3060] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:23.996522 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f97e5bad46343098418e8e34334b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.996773 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639cf3b7654a4ae896c6edd8b2ff1844. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:23.998428 773 raft_consensus.cc:3060] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:24.001791 2143 consensus_queue.cc:1048] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:24.006515 878 tablet_service.cc:1505] Processing CreateTablet for tablet 47f610e6d08046a385c58274abab0e0b (DEFAULT_TABLE table=default.Table_2be03ca8d3964e4db04c188da2e59bd3 [id=1eeba8e8c0be4ac2b904a59363a89e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.006986 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f610e6d08046a385c58274abab0e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.006923 884 tablet_service.cc:1505] Processing CreateTablet for tablet 1c5ee4114cb34540af2c5b332ebe82ab (DEFAULT_TABLE table=default.Table_0f9951606583457cb248ab0e617b466b [id=3a0ead3ea5aa4cdea9e6d8fc156bee96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.007599 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5ee4114cb34540af2c5b332ebe82ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.009032 879 tablet_service.cc:1505] Processing CreateTablet for tablet 0634838779854905b64dcdd028deb1e1 (DEFAULT_TABLE table=default.Table_5a79c363f13a4849a6be94ea935e5b89 [id=4093d9a107134e4b90970ddecc931769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.009390 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0634838779854905b64dcdd028deb1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.010134 880 tablet_service.cc:1505] Processing CreateTablet for tablet d5e0b89d02c04625b8e31910d60d5d40 (DEFAULT_TABLE table=default.Table_b72c21c5b00d430ea93eff8bf8bcb228 [id=4d54f97a2cde4143a185eb4af35bb07d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.010524 882 tablet_service.cc:1505] Processing CreateTablet for tablet 639cf3b7654a4ae896c6edd8b2ff1844 (DEFAULT_TABLE table=default.Table_53e58d3cdbfe4ab2945a18979051d7a2 [id=02c00037bc944c00b0124d72af62e19b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.010838 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5e0b89d02c04625b8e31910d60d5d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.010888 885 tablet_service.cc:1505] Processing CreateTablet for tablet 2f97e5bad46343098418e8e34334b892 (DEFAULT_TABLE table=default.Table_ec9d2344b1bc430784fe937726619f77 [id=eedcb23f130b4d9b8789d27f24ccf9b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.011284 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639cf3b7654a4ae896c6edd8b2ff1844. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.013278 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 3bed2c326906486e99300e05033c9c0f
I20251214 20:52:24.014573 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 639cf3b7654a4ae896c6edd8b2ff1844. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.015410 887 tablet_service.cc:1505] Processing CreateTablet for tablet 016581519f4c4e0d832b49346d55b41c (DEFAULT_TABLE table=default.Table_d6384d1f74bc4ec9a1985c5d8655e4b6 [id=9afdb569356648bb84089b1738dfc359]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.015759 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 016581519f4c4e0d832b49346d55b41c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.016615 2167 raft_consensus.cc:493] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:24.016772 2167 raft_consensus.cc:515] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.017552 2167 leader_election.cc:290] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:24.004884 1203 tablet_bootstrap.cc:492] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:24.021633 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bed2c326906486e99300e05033c9c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:24.013676 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 016581519f4c4e0d832b49346d55b41c (DEFAULT_TABLE table=default.Table_d6384d1f74bc4ec9a1985c5d8655e4b6 [id=9afdb569356648bb84089b1738dfc359]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.022552 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 016581519f4c4e0d832b49346d55b41c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.023373 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 1d7aa508b11944aa8e268741c1998a9e
I20251214 20:52:24.023478 1203 tablet_bootstrap.cc:654] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:24.024533 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7aa508b11944aa8e268741c1998a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:24.014072 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680 (DEFAULT_TABLE table=default.Table_d46e98eafc574e9a92bd66399d44b0fb [id=786a05478c234774b2b327f73d3c0cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.027254 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.028739 2184 consensus_queue.cc:1048] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:24.029682 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "012b81223d0f4ae9b742014e329f11a9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:24.029958 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7aa508b11944aa8e268741c1998a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:24.030570 989 leader_election.cc:343] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.031108 748 tablet_service.cc:1505] Processing CreateTablet for tablet a9c383e343544c83ada89bea6ab20e94 (DEFAULT_TABLE table=default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.031567 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c383e343544c83ada89bea6ab20e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.034677 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bed2c326906486e99300e05033c9c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:24.035248 751 tablet_service.cc:1505] Processing CreateTablet for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680 (DEFAULT_TABLE table=default.Table_d46e98eafc574e9a92bd66399d44b0fb [id=786a05478c234774b2b327f73d3c0cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.035629 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.035853 1027 tablet_service.cc:1505] Processing CreateTablet for tablet a9c383e343544c83ada89bea6ab20e94 (DEFAULT_TABLE table=default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.036338 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c383e343544c83ada89bea6ab20e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.039466 1021 tablet_service.cc:1505] Processing CreateTablet for tablet a41ed28272ad4e27bd2203dc4d3ad113 (DEFAULT_TABLE table=default.Table_e552103bd07e4f68b25576d6044d8861 [id=80a316658e6d494387628a03406f6480]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.039889 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a41ed28272ad4e27bd2203dc4d3ad113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.041214 1022 tablet_service.cc:1505] Processing CreateTablet for tablet ad99467157df4c37bb9b518244a5dbd8 (DEFAULT_TABLE table=default.Table_e125108df3c94234bedbdf467db81f28 [id=a47338e44a9e4642baa8407af70be735]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.041555 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad99467157df4c37bb9b518244a5dbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.041941 1982 ts_tablet_manager.cc:1929] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:24.042050 1982 log.cc:1199] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/c2ae9dc8ead8403cb7215af8778872f0
I20251214 20:52:24.042315 1982 ts_tablet_manager.cc:1950] T c2ae9dc8ead8403cb7215af8778872f0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:24.043555 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5275181d46dc4a85b27fc0c45fa587fe" } modify_external_catalogs: true
I20251214 20:52:24.044531 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5275181d46dc4a85b27fc0c45fa587fe" } modify_external_catalogs: true
I20251214 20:52:24.044829 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 163) took 1867 ms. Trace:
I20251214 20:52:24.045006 750 tablet_service.cc:1505] Processing CreateTablet for tablet a41ed28272ad4e27bd2203dc4d3ad113 (DEFAULT_TABLE table=default.Table_e552103bd07e4f68b25576d6044d8861 [id=80a316658e6d494387628a03406f6480]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.045499 738 tablet_service.cc:1505] Processing CreateTablet for tablet ad99467157df4c37bb9b518244a5dbd8 (DEFAULT_TABLE table=default.Table_e125108df3c94234bedbdf467db81f28 [id=a47338e44a9e4642baa8407af70be735]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.045686 1982 rpcz_store.cc:276] 1214 20:52:22.177651 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.177740 (+ 89us) service_pool.cc:225] Handling call
1214 20:52:23.963612 (+1785872us) ts_tablet_manager.cc:1151] Deleting tablet c2ae9dc8ead8403cb7215af8778872f0
1214 20:52:23.963621 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.994854 (+ 31233us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.041817 (+ 46963us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.044812 (+ 2995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1776112}
I20251214 20:52:24.045964 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad99467157df4c37bb9b518244a5dbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.046041 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 164) took 1866 ms. Trace:
I20251214 20:52:24.046096 1982 rpcz_store.cc:276] 1214 20:52:22.179371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.179445 (+ 74us) service_pool.cc:225] Handling call
1214 20:52:24.045953 (+1866508us) ts_tablet_manager.cc:1151] Deleting tablet 8af99e0bd8774a678575847a9ec2e557
1214 20:52:24.045961 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.046027 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1858408}
I20251214 20:52:24.046329 1982 tablet_replica.cc:333] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:24.048595 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 6b389a63a58d438db41a343db6e04a88 (DEFAULT_TABLE table=default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.049201 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b389a63a58d438db41a343db6e04a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.049654 1982 raft_consensus.cc:2243] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:24.049974 1021 tablet_service.cc:1505] Processing CreateTablet for tablet b87b0871c2454971a15dd5643c155694 (DEFAULT_TABLE table=default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.050006 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_cf7fd4bd0b764cc79acc374ea7bb5187" } new_table_name: "default.Table_c5bdb81886bf4361af53fd1a3b3b0b16" modify_external_catalogs: true
I20251214 20:52:24.050426 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87b0871c2454971a15dd5643c155694. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.050791 1982 raft_consensus.cc:2272] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.052685 1982 ts_tablet_manager.cc:1916] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:24.052752 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a41ed28272ad4e27bd2203dc4d3ad113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.055199 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_dd97baf658ea4f44bba79f081b5ff75e" } new_table_name: "default.Table_1ba7876ea93447038c0de7f37fb52825" modify_external_catalogs: true
I20251214 20:52:24.055567 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_7660878c81e04c17b9b6fa8d61ef43a6" } modify_external_catalogs: true
I20251214 20:52:24.055979 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_f1a942f776f74d61922b38c19e43f5df" } new_table_name: "default.Table_fefd2aebe38641a6892916b197d27abc" modify_external_catalogs: true
I20251214 20:52:24.056465 744 tablet_service.cc:1505] Processing CreateTablet for tablet 016581519f4c4e0d832b49346d55b41c (DEFAULT_TABLE table=default.Table_d6384d1f74bc4ec9a1985c5d8655e4b6 [id=9afdb569356648bb84089b1738dfc359]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.057250 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_eb4f37e15d9a4b1b9e3892c7891329bc" } new_table_name: "default.Table_2534df78188c4a16b12a732223bb9aaa" modify_external_catalogs: true
I20251214 20:52:24.058395 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_4e0d19a09c1f44eb85c4a776f6cd599a" } new_table_name: "default.Table_b14d897359c140bab6d04403fd59f31a" modify_external_catalogs: true
I20251214 20:52:24.058606 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 016581519f4c4e0d832b49346d55b41c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.063851 743 tablet_service.cc:1505] Processing CreateTablet for tablet b87b0871c2454971a15dd5643c155694 (DEFAULT_TABLE table=default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.064759 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87b0871c2454971a15dd5643c155694. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.065457 454 master_service.cc:954] ReplaceTablet: received request to replace tablet 47e813373fbd482298312ead4f037c01 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:24.073288 1982 ts_tablet_manager.cc:1929] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:24.077472 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bed2c326906486e99300e05033c9c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.078215 1982 log.cc:1199] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/3e5e220d083243eb8aa4328f366cd7ff
I20251214 20:52:24.081419 2143 raft_consensus.cc:493] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:24.081611 2143 raft_consensus.cc:515] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.082399 2143 leader_election.cc:290] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:24.082873 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f97e5bad46343098418e8e34334b892. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.083366 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_0fb8f39742d245ac9d20fc7d6bf7e7c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:24.083573 2184 consensus_queue.cc:1048] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:24.084002 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fb8f39742d245ac9d20fc7d6bf7e7c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.084656 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_bf73b75ea64f496da60f3a9a6afa5d6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:24.084836 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f900ebe92efa476cac232131a818d449" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:24.085304 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf73b75ea64f496da60f3a9a6afa5d6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.085530 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f900ebe92efa476cac232131a818d449" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:24.086128 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_680d6597d5f0419e83bfca87ec6d7787"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:24.086385 1203 tablet_bootstrap.cc:492] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:24.086517 1203 ts_tablet_manager.cc:1403] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.000s
W20251214 20:52:24.086573 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_680d6597d5f0419e83bfca87ec6d7787 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.087323 1203 raft_consensus.cc:359] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.087970 1203 raft_consensus.cc:385] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:24.088349 1203 raft_consensus.cc:740] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:24.088303 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "012b81223d0f4ae9b742014e329f11a9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:24.088882 1203 consensus_queue.cc:260] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.090469 881 tablet_service.cc:1552] Processing DeleteTablet for tablet bc73d590531c409e9f60670435dc8223 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.090776 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e5e220d083243eb8aa4328f366cd7ff with delete_type TABLET_DATA_DELETED (replaced by tablet 6a000951f61948ac85fdd431f3d162c2) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.090903 894 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24e4f4e74040249068505058c15622 with delete_type TABLET_DATA_DELETED (replaced by tablet 0dbfe8ee68bf4a2c88489b1f81b36991) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.091038 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 16816fa6b8c34207bb929147910b93cb with delete_type TABLET_DATA_DELETED (replaced by tablet 532c44785f204f988518c6fd628f76ca) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.091370 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 274185a4115b458e8e9a376bdbcc870b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.091393 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 8af99e0bd8774a678575847a9ec2e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:12 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.092618 1203 ts_tablet_manager.cc:1434] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20251214 20:52:24.094905 848 leader_election.cc:343] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.098554 1203 tablet_bootstrap.cc:492] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:24.100672 851 leader_election.cc:343] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.100986 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7aa508b11944aa8e268741c1998a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.101166 915 raft_consensus.cc:2468] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:24.101562 991 leader_election.cc:304] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:24.101862 2184 raft_consensus.cc:2804] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:24.101950 2184 raft_consensus.cc:493] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:24.102015 2184 raft_consensus.cc:3060] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:24.102501 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d164747cb294bd4b126b4420231867d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.102752 851 leader_election.cc:304] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:24.103832 894 tablet_service.cc:1552] Processing DeleteTablet for tablet a063083f54ad49d2a842fa080d44ed85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:13 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.104372 894 tablet_service.cc:1505] Processing CreateTablet for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680 (DEFAULT_TABLE table=default.Table_d46e98eafc574e9a92bd66399d44b0fb [id=786a05478c234774b2b327f73d3c0cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.104773 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.105535 2196 raft_consensus.cc:2749] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:24.105747 2196 raft_consensus.cc:493] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 074399864fb2470fa9e2ee87c1fe2212)
I20251214 20:52:24.105877 2196 raft_consensus.cc:515] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:24.106070 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b47ca1a03944cf1847d3578856dafa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.106432 895 tablet_service.cc:1505] Processing CreateTablet for tablet a41ed28272ad4e27bd2203dc4d3ad113 (DEFAULT_TABLE table=default.Table_e552103bd07e4f68b25576d6044d8861 [id=80a316658e6d494387628a03406f6480]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.106572 2196 leader_election.cc:290] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:24.106765 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a41ed28272ad4e27bd2203dc4d3ad113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.107499 1769 fs_manager.cc:714] Time spent opening directory manager: real 0.171s user 0.008s sys 0.000s
I20251214 20:52:24.107643 883 tablet_service.cc:1505] Processing CreateTablet for tablet ad99467157df4c37bb9b518244a5dbd8 (DEFAULT_TABLE table=default.Table_e125108df3c94234bedbdf467db81f28 [id=a47338e44a9e4642baa8407af70be735]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.107721 2184 raft_consensus.cc:515] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.107896 882 tablet_service.cc:1505] Processing CreateTablet for tablet a9c383e343544c83ada89bea6ab20e94 (DEFAULT_TABLE table=default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.108000 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad99467157df4c37bb9b518244a5dbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.108203 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c383e343544c83ada89bea6ab20e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.108338 2184 leader_election.cc:290] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:24.108484 890 tablet_service.cc:1505] Processing CreateTablet for tablet b87b0871c2454971a15dd5643c155694 (DEFAULT_TABLE table=default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.108788 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b87b0871c2454971a15dd5643c155694. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:24.109447 848 leader_election.cc:343] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: ba50e32f7e724868bb891045a939e362
I20251214 20:52:24.109582 891 tablet_service.cc:1505] Processing CreateTablet for tablet 6b389a63a58d438db41a343db6e04a88 (DEFAULT_TABLE table=default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.109941 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b389a63a58d438db41a343db6e04a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.110004 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "012b81223d0f4ae9b742014e329f11a9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:24.110198 904 raft_consensus.cc:3060] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:24.112296 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_7660878c81e04c17b9b6fa8d61ef43a6" } modify_external_catalogs: true
I20251214 20:52:24.112768 884 tablet_service.cc:1552] Processing DeleteTablet for tablet f342dafae1d04be3ba38f7c0e4f5d058 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.114152 881 tablet_service.cc:1552] Processing DeleteTablet for tablet eb29fd8c85344ed8aa8d880e2ed240bc with delete_type TABLET_DATA_DELETED (replaced by tablet 7242be189ca54c9182181e33d6743e6b) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:24.117235 851 leader_election.cc:343] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: ba50e32f7e724868bb891045a939e362
I20251214 20:52:24.118789 2279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251214 20:52:24.119827 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ac06f852fc4abea5cb213926002ca1 with delete_type TABLET_DATA_DELETED (replaced by tablet 248e3666fd13434e97378eb5a98dfadd) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.121683 1769 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.009s sys 0.000s
I20251214 20:52:24.122344 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet c2ae9dc8ead8403cb7215af8778872f0 (table default.Table_2342c4af4f9b420b9b7cbcf327ce1733 [id=976ca3fb3a164669abeff0d8eb137e41]) successfully deleted
W20251214 20:52:24.124331 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8af99e0bd8774a678575847a9ec2e557 because the tablet was not found. No further retry: Not found: Tablet not found: 8af99e0bd8774a678575847a9ec2e557
I20251214 20:52:24.123735 1769 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal
uuid: "e6713a20a82b434f8a8bc122113e26ad"
format_stamp: "Formatted at 2025-12-14 20:52:05 on dist-test-slave-njhz"
I20251214 20:52:24.125317 1769 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20251214 20:52:24.130051 448 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43fe3bb1ed3941ce9078376f8f478408 consensus state has no leader
I20251214 20:52:24.131266 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_19a53df5559b4eb5bfc7b41b54be081b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:24.132251 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19a53df5559b4eb5bfc7b41b54be081b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:24.131619 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e90d11d69b8d4fd49150aa67a8190e5b consensus state has no leader
I20251214 20:52:24.133770 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_3474e4ca80964e3b8b7b66e1fde4815f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:24.134423 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3474e4ca80964e3b8b7b66e1fde4815f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.136317 1203 tablet_bootstrap.cc:654] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:24.140187 892 tablet_service.cc:1552] Processing DeleteTablet for tablet ca263a80702f4f64aa7c872c2be08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.144620 1982 ts_tablet_manager.cc:1950] T 3e5e220d083243eb8aa4328f366cd7ff P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:24.145119 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 165) took 1963 ms. Trace:
I20251214 20:52:24.145274 1982 rpcz_store.cc:276] 1214 20:52:22.181356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.181430 (+ 74us) service_pool.cc:225] Handling call
1214 20:52:24.046273 (+1864843us) ts_tablet_manager.cc:1151] Deleting tablet 3e5e220d083243eb8aa4328f366cd7ff
1214 20:52:24.046280 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.061779 (+ 15499us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.073181 (+ 11402us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.145101 (+ 71920us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1858669}
I20251214 20:52:24.145802 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 166) took 1964 ms. Trace:
I20251214 20:52:24.145961 1203 tablet_bootstrap.cc:492] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:24.145920 1982 rpcz_store.cc:276] 1214 20:52:22.181748 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.181961 (+ 213us) service_pool.cc:225] Handling call
1214 20:52:24.145641 (+1963680us) ts_tablet_manager.cc:1151] Deleting tablet 52daf1204bf142e7b82852bdec6582fb
1214 20:52:24.145651 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.145726 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1957983}
I20251214 20:52:24.146162 1982 tablet_replica.cc:333] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:24.146359 1982 raft_consensus.cc:2243] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:24.146495 1982 raft_consensus.cc:2272] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.147526 1203 ts_tablet_manager.cc:1403] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20251214 20:52:24.149573 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 3e5e220d083243eb8aa4328f366cd7ff (table default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]) successfully deleted
W20251214 20:52:24.149893 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 52daf1204bf142e7b82852bdec6582fb because the tablet was not found. No further retry: Not found: Tablet not found: 52daf1204bf142e7b82852bdec6582fb
I20251214 20:52:24.150004 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "012b81223d0f4ae9b742014e329f11a9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:24.150820 1982 ts_tablet_manager.cc:1916] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:24.150799 744 tablet_service.cc:1505] Processing CreateTablet for tablet 6b389a63a58d438db41a343db6e04a88 (DEFAULT_TABLE table=default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:24.153920 989 leader_election.cc:343] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.155447 851 leader_election.cc:304] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:24.162810 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e1835374cfa48939d72476870accc3b consensus state has no leader
I20251214 20:52:24.163457 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_4cb054a62c404f1a952f39810b6d12f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:24.163909 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d6683a16a95f4274b82ff74dc7e73597"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:24.164068 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cb054a62c404f1a952f39810b6d12f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:24.164398 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6683a16a95f4274b82ff74dc7e73597 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.165686 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_828c0037b27e439a9dbb15886bb51344" } modify_external_catalogs: true
I20251214 20:52:24.165846 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_828c0037b27e439a9dbb15886bb51344" } modify_external_catalogs: true
I20251214 20:52:24.167775 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet feb19ea389af4ea6bb158a0e8aaeb094
I20251214 20:52:24.168556 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_4278bf82042b47c2b4d0e799f115c52f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:24.169086 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4278bf82042b47c2b4d0e799f115c52f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.169505 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_1191163db85341c5b41c4baf61bc8df5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:24.169947 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1191163db85341c5b41c4baf61bc8df5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.170724 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet feb19ea389af4ea6bb158a0e8aaeb094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:24.162632 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_0a325ff1f4634ce7ada593df8f5a9fd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:24.171825 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a325ff1f4634ce7ada593df8f5a9fd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:24.171476 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79e621f29c01448399d22615284fccd4 consensus state has no leader
I20251214 20:52:24.174713 1203 raft_consensus.cc:359] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.175269 738 tablet_service.cc:1552] Processing DeleteTablet for tablet feb19ea389af4ea6bb158a0e8aaeb094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:24.178951 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80098b7c24824d129b8f25daf2115460 consensus state has no leader
I20251214 20:52:24.179822 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet ac64f4fcb6a943668584658df19f950c
I20251214 20:52:24.180581 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet ac64f4fcb6a943668584658df19f950c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:24.183791 1203 raft_consensus.cc:385] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:24.183784 888 tablet_service.cc:1552] Processing DeleteTablet for tablet b315744c8cd043c5a8bdceec73a7e28b with delete_type TABLET_DATA_DELETED (replaced by tablet f945814952b14131ba4e0c67a403a680) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.184405 1203 raft_consensus.cc:740] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:24.185981 1202 tablet_bootstrap.cc:654] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:24.186666 989 consensus_peers.cc:597] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e16fe751f10f4616a00d0af811998729. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:24.185117 1203 consensus_queue.cc:260] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.190755 1203 ts_tablet_manager.cc:1434] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20251214 20:52:24.193672 1203 tablet_bootstrap.cc:492] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:24.195858 1203 tablet_bootstrap.cc:654] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:24.196362 1977 ts_tablet_manager.cc:1929] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:24.217857 446 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_816a237b5485489c9dc522f872701944" } modify_external_catalogs: true
I20251214 20:52:24.218071 446 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_816a237b5485489c9dc522f872701944" } modify_external_catalogs: true
I20251214 20:52:24.219062 2143 raft_consensus.cc:2749] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251214 20:52:24.211562 848 consensus_peers.cc:597] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:24.225838 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_2534df78188c4a16b12a732223bb9aaa" } modify_external_catalogs: true
I20251214 20:52:24.226111 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_2534df78188c4a16b12a732223bb9aaa" } modify_external_catalogs: true
I20251214 20:52:24.226333 879 tablet_service.cc:1552] Processing DeleteTablet for tablet feb19ea389af4ea6bb158a0e8aaeb094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.226538 888 tablet_service.cc:1552] Processing DeleteTablet for tablet ac64f4fcb6a943668584658df19f950c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.226646 1977 log.cc:1199] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/2a7fc15882e2474bb6885070be13b1fa
W20251214 20:52:24.227561 440 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 80098b7c24824d129b8f25daf2115460 (table default.Table_b14d897359c140bab6d04403fd59f31a [id=2a055be9028342be97f5c98f03f5a137], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 80098b7c24824d129b8f25daf2115460 consensus state has no leader
I20251214 20:52:24.230281 1856 ts_tablet_manager.cc:1929] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251214 20:52:24.232056 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 72678.4us
I20251214 20:52:24.232787 2143 raft_consensus.cc:493] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:24.236480 2143 raft_consensus.cc:515] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.237141 2143 leader_election.cc:290] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:24.237398 1977 ts_tablet_manager.cc:1950] T 2a7fc15882e2474bb6885070be13b1fa P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:24.244304 904 raft_consensus.cc:2468] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:24.245834 2231 raft_consensus.cc:493] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63848a9fb78a4013a6f1657621272d75)
I20251214 20:52:24.246290 2231 raft_consensus.cc:515] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.247119 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 183) took 1256 ms. Trace:
I20251214 20:52:24.247320 1977 rpcz_store.cc:276] 1214 20:52:22.988526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.991598 (+ 3072us) service_pool.cc:225] Handling call
1214 20:52:23.947704 (+956106us) ts_tablet_manager.cc:1151] Deleting tablet 2a7fc15882e2474bb6885070be13b1fa
1214 20:52:23.947717 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:23.953214 (+ 5497us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.118999 (+165785us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.244864 (+125865us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30,"tablet-delete.queue_time_us":955970}
I20251214 20:52:24.247790 1977 tablet_replica.cc:333] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:24.248032 1977 raft_consensus.cc:2243] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:24.248512 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 2a7fc15882e2474bb6885070be13b1fa (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]) successfully deleted
I20251214 20:52:24.248528 1856 log.cc:1199] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ca263a80702f4f64aa7c872c2be08207
I20251214 20:52:24.247798 2231 leader_election.cc:290] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:24.248852 1977 raft_consensus.cc:2272] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.235499 1203 tablet_bootstrap.cc:492] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:24.249075 1203 ts_tablet_manager.cc:1403] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.000s
I20251214 20:52:24.249660 1977 ts_tablet_manager.cc:1916] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:24.250164 1203 raft_consensus.cc:359] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.250344 1203 raft_consensus.cc:385] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:24.250389 1203 raft_consensus.cc:740] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:24.250599 1203 consensus_queue.cc:260] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:24.252367 713 consensus_peers.cc:597] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fcfcfe81fe24d369eea696efc61d21c. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:24.254385 1203 ts_tablet_manager.cc:1434] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:24.254818 1203 tablet_bootstrap.cc:492] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:24.258309 1856 ts_tablet_manager.cc:1950] T ca263a80702f4f64aa7c872c2be08207 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:24.268810 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "242a77ae57174de48f4eaf43384ee0ba" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:24.269699 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b389a63a58d438db41a343db6e04a88. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:24.276544 848 leader_election.cc:343] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 4b86f4adaf4540f48a4f42bc45e259e5
W20251214 20:52:24.276952 848 leader_election.cc:343] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.280865 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 43fe3bb1ed3941ce9078376f8f478408
I20251214 20:52:24.283175 446 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 054cfd587d19409fbdf7f4848bcaf3f9
I20251214 20:52:24.285719 455 catalog_manager.cc:5654] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:24.286841 455 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (table default.Table_89c0478b97144f35b1a0adeb64be2d9f [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]). Expected version 1 got 0
I20251214 20:52:24.288519 894 tablet_service.cc:1505] Processing CreateTablet for tablet 197d3d340f6e4cbea0d621907b8c76d4 (DEFAULT_TABLE table=default.Table_f4c54cf75bea445185ff40f822983fb5 [id=12f54ade07c6481298fe1a6783f719d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.290294 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 197d3d340f6e4cbea0d621907b8c76d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.291006 883 tablet_service.cc:1505] Processing CreateTablet for tablet 52c05204502a45e18c2b862e21c6f361 (DEFAULT_TABLE table=default.Table_9f0b2a3e0ef1496db7886462be5513c3 [id=4cc99db976364563b300df561cf37aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.291474 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c05204502a45e18c2b862e21c6f361. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.292958 895 tablet_service.cc:1505] Processing CreateTablet for tablet 2807af0754914688bd7a090b1bc32eb1 (DEFAULT_TABLE table=default.Table_5d1ff53eee3c4eb596c1b728e7d3bc1d [id=9d1473df58ee486aa6a55893aca9d54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.295461 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 727e7b61ae3942098712720d344c5310
I20251214 20:52:24.303930 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 43fe3bb1ed3941ce9078376f8f478408 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:24.307825 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 054cfd587d19409fbdf7f4848bcaf3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.309451 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 43fe3bb1ed3941ce9078376f8f478408 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.310277 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2807af0754914688bd7a090b1bc32eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.319923 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 727e7b61ae3942098712720d344c5310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.320755 892 tablet_service.cc:1505] Processing CreateTablet for tablet 15b3216f3d2b443e8ab2f994445956f9 (DEFAULT_TABLE table=default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.320958 880 tablet_service.cc:1505] Processing CreateTablet for tablet 4f358df4d9414dcdb0d197f434e28118 (DEFAULT_TABLE table=default.Table_d8bb40c2324c4c0eb7e52aa96e920813 [id=3b04d86591b14695a334b68b0572f371]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.321187 879 tablet_service.cc:1505] Processing CreateTablet for tablet fa141b2570514c6aacb2850e4bfd1869 (DEFAULT_TABLE table=default.Table_47baf32513e6487fbd4ffde9bcba97ca [id=5ea1e541e5454f1ca96338573fda5705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.321383 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15b3216f3d2b443e8ab2f994445956f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.321623 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f358df4d9414dcdb0d197f434e28118. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.321990 878 tablet_service.cc:1505] Processing CreateTablet for tablet 5ec7824f5eff4cc59c45acd3b3b3d194 (DEFAULT_TABLE table=default.Table_1e4dafc4ff8940ea90956623c915e3fd [id=be1be47d388347e3a6b7a80105688be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.322316 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec7824f5eff4cc59c45acd3b3b3d194. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.320958 888 tablet_service.cc:1505] Processing CreateTablet for tablet 2593f67c7c6441b694d99286e3ba5883 (DEFAULT_TABLE table=default.Table_bde8bb1617d3450f999e9a7960d96641 [id=3a3dca8b90264d949ac9bead0b0743fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.324083 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2593f67c7c6441b694d99286e3ba5883. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.327173 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa141b2570514c6aacb2850e4bfd1869. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.328056 890 tablet_service.cc:1505] Processing CreateTablet for tablet 2398f176ee044c15841eaa40cb74974c (DEFAULT_TABLE table=default.Table_32323ff67ecd4fe38fa02169904b88bf [id=b1ee89b437fb42a4b5e2c3d386ee0284]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.328545 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2398f176ee044c15841eaa40cb74974c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.333369 882 tablet_service.cc:1505] Processing CreateTablet for tablet ba548574d8784701974e611d7a9ad9b1 (DEFAULT_TABLE table=default.Table_0c936131512c4d99b3999d74fabf18d3 [id=1c5148c58b1c4fe39bac0697004ca484]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.334008 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 054cfd587d19409fbdf7f4848bcaf3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:24.334295 749 tablet_service.cc:1505] Processing CreateTablet for tablet 52c05204502a45e18c2b862e21c6f361 (DEFAULT_TABLE table=default.Table_9f0b2a3e0ef1496db7886462be5513c3 [id=4cc99db976364563b300df561cf37aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.335505 750 tablet_service.cc:1505] Processing CreateTablet for tablet 197d3d340f6e4cbea0d621907b8c76d4 (DEFAULT_TABLE table=default.Table_f4c54cf75bea445185ff40f822983fb5 [id=12f54ade07c6481298fe1a6783f719d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.335690 751 tablet_service.cc:1552] Processing DeleteTablet for tablet ac64f4fcb6a943668584658df19f950c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:24.335618 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3f7396ac07d649fa8d342ae05765da80" } new_table_name: "default.Table_13184fd4bbc74e5f8bea847a1c8674eb" modify_external_catalogs: true
I20251214 20:52:24.340380 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_67fa945ac69a4a599f3dd43cfa93c0ff" } new_table_name: "default.Table_ed823cd2c6d94ed2b4707f427c8a2867" modify_external_catalogs: true
I20251214 20:52:24.344466 1769 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251214 20:52:24.346444 1769 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20251214 20:52:24.346825 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1eff8a23d48c43e2bdbfb34f4bab6edf consensus state has no leader
I20251214 20:52:24.346946 1769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251214 20:52:24.347218 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c05204502a45e18c2b862e21c6f361. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.348084 1206 tablet_bootstrap.cc:492] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:24.348254 1206 ts_tablet_manager.cc:1403] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.373s user 0.004s sys 0.014s
I20251214 20:52:24.353739 2155 raft_consensus.cc:493] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:24.353938 2155 raft_consensus.cc:515] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.354796 2155 leader_election.cc:290] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:24.356182 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "242a77ae57174de48f4eaf43384ee0ba" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:24.360186 885 tablet_service.cc:1505] Processing CreateTablet for tablet f98be57ce1b14f20a3afcb79979b6c22 (DEFAULT_TABLE table=default.Table_da414f75240a40e6b20b644312217eae [id=0ff4524a6b97452fb0a8dce29d18a44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.360462 753 tablet_service.cc:1505] Processing CreateTablet for tablet 2807af0754914688bd7a090b1bc32eb1 (DEFAULT_TABLE table=default.Table_5d1ff53eee3c4eb596c1b728e7d3bc1d [id=9d1473df58ee486aa6a55893aca9d54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.360499 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 169) took 3219 ms. Trace:
I20251214 20:52:24.360514 891 tablet_service.cc:1505] Processing CreateTablet for tablet c3453fc996fb4d2b8cf0f80152098c43 (DEFAULT_TABLE table=default.Table_9196cd658f9041269835197751563391 [id=2f57c28c9c624a6ca1986c8831b85169]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.360621 1856 rpcz_store.cc:276] 1214 20:52:21.041089 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.041138 (+ 49us) service_pool.cc:225] Handling call
1214 20:52:23.729643 (+2688505us) ts_tablet_manager.cc:1151] Deleting tablet ca263a80702f4f64aa7c872c2be08207
1214 20:52:23.729651 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.226382 (+496731us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.230145 (+ 3763us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.260246 (+ 30101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2688207}
I20251214 20:52:24.361094 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 170) took 3277 ms. Trace:
I20251214 20:52:24.360968 1206 raft_consensus.cc:359] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.361176 1206 raft_consensus.cc:385] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:24.361227 1206 raft_consensus.cc:740] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:24.361161 1856 rpcz_store.cc:276] 1214 20:52:21.083750 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.083807 (+ 57us) service_pool.cc:225] Handling call
1214 20:52:24.360955 (+3277148us) ts_tablet_manager.cc:1151] Deleting tablet c4b22043d81b493fb12116422b1d3fb1
1214 20:52:24.360968 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.361078 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3276731}
I20251214 20:52:24.361399 1856 tablet_replica.cc:333] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:24.361601 1856 raft_consensus.cc:2243] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:24.361783 1856 raft_consensus.cc:2272] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.362474 1856 ts_tablet_manager.cc:1916] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:24.363341 2167 raft_consensus.cc:493] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:24.363502 2167 raft_consensus.cc:515] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.361428 1206 consensus_queue.cc:260] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.364115 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2807af0754914688bd7a090b1bc32eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.364132 2167 leader_election.cc:290] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:24.364667 1206 ts_tablet_manager.cc:1434] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251214 20:52:24.364987 1206 tablet_bootstrap.cc:492] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:24.368441 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 197d3d340f6e4cbea0d621907b8c76d4 (DEFAULT_TABLE table=default.Table_f4c54cf75bea445185ff40f822983fb5 [id=12f54ade07c6481298fe1a6783f719d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.369912 447 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5879a88313cb488b8092323f94dfd3a7" } new_table_name: "default.Table_3a068835084e406eb4047a827faeb5d7" modify_external_catalogs: true
I20251214 20:52:24.370162 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_58619bc0f4b54a208cabd270329ccf82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:24.370913 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_9ba7f3a5c10e401b9edd530f37db3e17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:24.371390 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_01b1f692968d4dbe8d92746b6017a8b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:24.371819 454 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
W20251214 20:52:24.372282 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ba7f3a5c10e401b9edd530f37db3e17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.372720 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_db687aab85f2411f8433eb14724da9ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:24.373116 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db687aab85f2411f8433eb14724da9ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.373929 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 197d3d340f6e4cbea0d621907b8c76d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:24.371825 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01b1f692968d4dbe8d92746b6017a8b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.375687 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_bb0aa14e61414a5c95c2923827b7cae7" } new_table_name: "default.Table_affc01186d6d429bb07259ca92056991" modify_external_catalogs: true
I20251214 20:52:24.376895 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 727e7b61ae3942098712720d344c5310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:24.377187 1203 tablet_bootstrap.cc:654] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:24.377324 742 tablet_service.cc:1505] Processing CreateTablet for tablet fa141b2570514c6aacb2850e4bfd1869 (DEFAULT_TABLE table=default.Table_47baf32513e6487fbd4ffde9bcba97ca [id=5ea1e541e5454f1ca96338573fda5705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.377624 755 tablet_service.cc:1505] Processing CreateTablet for tablet 5ec7824f5eff4cc59c45acd3b3b3d194 (DEFAULT_TABLE table=default.Table_1e4dafc4ff8940ea90956623c915e3fd [id=be1be47d388347e3a6b7a80105688be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.377723 738 tablet_service.cc:1505] Processing CreateTablet for tablet c3453fc996fb4d2b8cf0f80152098c43 (DEFAULT_TABLE table=default.Table_9196cd658f9041269835197751563391 [id=2f57c28c9c624a6ca1986c8831b85169]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:24.375793 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58619bc0f4b54a208cabd270329ccf82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.378110 741 tablet_service.cc:1505] Processing CreateTablet for tablet 4f358df4d9414dcdb0d197f434e28118 (DEFAULT_TABLE table=default.Table_d8bb40c2324c4c0eb7e52aa96e920813 [id=3b04d86591b14695a334b68b0572f371]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.378110 754 tablet_service.cc:1505] Processing CreateTablet for tablet 51366a6aefd54077877ec5d076394d79 (DEFAULT_TABLE table=default.Table_1191163db85341c5b41c4baf61bc8df5 [id=fcb196b8ebb14e1785ea355e7d406b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.380132 757 tablet_service.cc:1505] Processing CreateTablet for tablet 2593f67c7c6441b694d99286e3ba5883 (DEFAULT_TABLE table=default.Table_bde8bb1617d3450f999e9a7960d96641 [id=3a3dca8b90264d949ac9bead0b0743fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.382194 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c845c10eeec54a7aa19cb8e0cebeb3a6" } new_table_name: "default.Table_fcafcf3c6ddb4457aea41320f7bf0b74" modify_external_catalogs: true
I20251214 20:52:24.383514 1025 tablet_service.cc:1505] Processing CreateTablet for tablet ba548574d8784701974e611d7a9ad9b1 (DEFAULT_TABLE table=default.Table_0c936131512c4d99b3999d74fabf18d3 [id=1c5148c58b1c4fe39bac0697004ca484]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.383623 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_1e0e46b1f1e1418eadafc528585358f3" } modify_external_catalogs: true
I20251214 20:52:24.383967 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 52c05204502a45e18c2b862e21c6f361 (DEFAULT_TABLE table=default.Table_9f0b2a3e0ef1496db7886462be5513c3 [id=4cc99db976364563b300df561cf37aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.384290 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 2807af0754914688bd7a090b1bc32eb1 (DEFAULT_TABLE table=default.Table_5d1ff53eee3c4eb596c1b728e7d3bc1d [id=9d1473df58ee486aa6a55893aca9d54c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.384457 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 43fe3bb1ed3941ce9078376f8f478408 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:24.384624 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 054cfd587d19409fbdf7f4848bcaf3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:24.385054 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_1e0e46b1f1e1418eadafc528585358f3" } modify_external_catalogs: true
I20251214 20:52:24.386286 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba548574d8784701974e611d7a9ad9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.391513 746 tablet_service.cc:1505] Processing CreateTablet for tablet 15b3216f3d2b443e8ab2f994445956f9 (DEFAULT_TABLE table=default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.391947 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c05204502a45e18c2b862e21c6f361. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.395368 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2807af0754914688bd7a090b1bc32eb1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:24.396050 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4643effa71d4d18850913a59ae1c75d consensus state has no leader
I20251214 20:52:24.402208 1856 ts_tablet_manager.cc:1929] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:24.402309 747 tablet_service.cc:1505] Processing CreateTablet for tablet 6583412e5a1849c8b93c87ba7fdc4647 (DEFAULT_TABLE table=default.Table_3474e4ca80964e3b8b7b66e1fde4815f [id=586a58c36e9a4d4494f4538ebcb9ce36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.403189 756 tablet_service.cc:1505] Processing CreateTablet for tablet a88207c8cc5f4a6886de36c82c6b4c2f (DEFAULT_TABLE table=default.Table_bf73b75ea64f496da60f3a9a6afa5d6e [id=f41452b44a3c41db9752487fe11f201c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.408561 748 tablet_service.cc:1505] Processing CreateTablet for tablet 2398f176ee044c15841eaa40cb74974c (DEFAULT_TABLE table=default.Table_32323ff67ecd4fe38fa02169904b88bf [id=b1ee89b437fb42a4b5e2c3d386ee0284]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.411839 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_27e3193bffa748be95f7c923662ad899"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:24.412525 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27e3193bffa748be95f7c923662ad899 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.412868 739 tablet_service.cc:1505] Processing CreateTablet for tablet f891db00595b49ca9e6ae64721e4d744 (DEFAULT_TABLE table=default.Table_19a53df5559b4eb5bfc7b41b54be081b [id=203681807a4b40b0a1f69387cfecff38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.413256 745 tablet_service.cc:1505] Processing CreateTablet for tablet 6302145a6e7b4297a58d9d7f3ad02dce (DEFAULT_TABLE table=default.Table_680d6597d5f0419e83bfca87ec6d7787 [id=54e0182cd3b9461aa593a11aa7709002]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.414845 749 tablet_service.cc:1505] Processing CreateTablet for tablet f98be57ce1b14f20a3afcb79979b6c22 (DEFAULT_TABLE table=default.Table_da414f75240a40e6b20b644312217eae [id=0ff4524a6b97452fb0a8dce29d18a44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.415822 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6d9954b482a4427ba8a996c085f831bf" } modify_external_catalogs: true
I20251214 20:52:24.418066 881 tablet_service.cc:1505] Processing CreateTablet for tablet 51366a6aefd54077877ec5d076394d79 (DEFAULT_TABLE table=default.Table_1191163db85341c5b41c4baf61bc8df5 [id=fcb196b8ebb14e1785ea355e7d406b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.418440 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6d9954b482a4427ba8a996c085f831bf" } modify_external_catalogs: true
I20251214 20:52:24.420398 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ca263a80702f4f64aa7c872c2be08207 (table default.Table_642da83f7ed0444688afdf91b7a9e7e8 [id=bca3a2d37f8b4fc797a5c52a43c14869]) successfully deleted
I20251214 20:52:24.423000 1977 ts_tablet_manager.cc:1929] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:24.423183 1977 log.cc:1199] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/c4b22043d81b493fb12116422b1d3fb1
W20251214 20:52:24.423516 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c4b22043d81b493fb12116422b1d3fb1 because the tablet was not found. No further retry: Not found: Tablet not found: c4b22043d81b493fb12116422b1d3fb1
I20251214 20:52:24.423593 1977 ts_tablet_manager.cc:1950] T c4b22043d81b493fb12116422b1d3fb1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:24.424489 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 192) took 1432 ms. Trace:
I20251214 20:52:24.424585 1977 rpcz_store.cc:276] 1214 20:52:22.991559 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.991804 (+ 245us) service_pool.cc:225] Handling call
1214 20:52:24.247694 (+1255890us) ts_tablet_manager.cc:1151] Deleting tablet c4b22043d81b493fb12116422b1d3fb1
1214 20:52:24.247708 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.291939 (+ 44231us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.422825 (+130886us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.424437 (+ 1612us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":108,"tablet-delete.queue_time_us":1255770}
I20251214 20:52:24.425045 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_183ef00a0ed14492bb36f72ed1c73aa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:24.425330 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ae25c010c04548198844cd6f5f3a5693"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:24.425572 884 tablet_service.cc:1505] Processing CreateTablet for tablet 6302145a6e7b4297a58d9d7f3ad02dce (DEFAULT_TABLE table=default.Table_680d6597d5f0419e83bfca87ec6d7787 [id=54e0182cd3b9461aa593a11aa7709002]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:24.425653 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_183ef00a0ed14492bb36f72ed1c73aa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:24.425706 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae25c010c04548198844cd6f5f3a5693 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.426719 887 tablet_service.cc:1505] Processing CreateTablet for tablet 6583412e5a1849c8b93c87ba7fdc4647 (DEFAULT_TABLE table=default.Table_3474e4ca80964e3b8b7b66e1fde4815f [id=586a58c36e9a4d4494f4538ebcb9ce36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.427078 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_c96142a94af44f24bb7c6e51bf5e1f3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:24.427592 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c96142a94af44f24bb7c6e51bf5e1f3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.428777 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_e95c404ef11b4cf79b5a81ec1eddb57a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:24.429189 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e95c404ef11b4cf79b5a81ec1eddb57a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.433171 880 tablet_service.cc:1505] Processing CreateTablet for tablet 5e92f54daf4f495fb6f026c179852362 (DEFAULT_TABLE table=default.Table_d6683a16a95f4274b82ff74dc7e73597 [id=573bdb65191b4f09b43190844f2b0008]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:24.433679 442 consensus_peers.cc:597] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a -> Peer e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819): Couldn't send request to peer e6713a20a82b434f8a8bc122113e26ad. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34819: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20251214 20:52:24.435961 740 tablet_service.cc:1505] Processing CreateTablet for tablet 5e92f54daf4f495fb6f026c179852362 (DEFAULT_TABLE table=default.Table_d6683a16a95f4274b82ff74dc7e73597 [id=573bdb65191b4f09b43190844f2b0008]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.413164 752 tablet_service.cc:1505] Processing CreateTablet for tablet 71b8588773c442fa9add357455fc9d7d (DEFAULT_TABLE table=default.Table_4278bf82042b47c2b4d0e799f115c52f [id=2c037ca94c8a4c908d6354f2aa802bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.437711 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba548574d8784701974e611d7a9ad9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.438387 912 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "762068d560ff47918dd9fd8791083433" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:24.443387 877 tablet_service.cc:1505] Processing CreateTablet for tablet 5a43ee234b184275a6ab76b82fe5d283 (DEFAULT_TABLE table=default.Table_0a325ff1f4634ce7ada593df8f5a9fd6 [id=88a6486d16184397acd4ef205ca46a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.443565 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet c4b22043d81b493fb12116422b1d3fb1 (table default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]) successfully deleted
I20251214 20:52:24.443877 893 tablet_service.cc:1505] Processing CreateTablet for tablet a6df388bdfbc40c183c4b14af2336a6a (DEFAULT_TABLE table=default.Table_0fb8f39742d245ac9d20fc7d6bf7e7c3 [id=552c8b213ba64d4bbdd3363cdc6625ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.444909 889 tablet_service.cc:1505] Processing CreateTablet for tablet f891db00595b49ca9e6ae64721e4d744 (DEFAULT_TABLE table=default.Table_19a53df5559b4eb5bfc7b41b54be081b [id=203681807a4b40b0a1f69387cfecff38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.445202 886 tablet_service.cc:1505] Processing CreateTablet for tablet a88207c8cc5f4a6886de36c82c6b4c2f (DEFAULT_TABLE table=default.Table_bf73b75ea64f496da60f3a9a6afa5d6e [id=f41452b44a3c41db9752487fe11f201c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.445415 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98be57ce1b14f20a3afcb79979b6c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.447150 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3453fc996fb4d2b8cf0f80152098c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.447492 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51366a6aefd54077877ec5d076394d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.447589 912 raft_consensus.cc:2468] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
W20251214 20:52:24.449971 851 leader_election.cc:343] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.450150 851 leader_election.cc:304] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:24.451427 2286 raft_consensus.cc:2749] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:24.451519 1202 tablet_bootstrap.cc:492] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:24.459106 447 master_service.cc:954] ReplaceTablet: received request to replace tablet 57f2314d33fa474ca36dafe0a0eae5ae from {username='slave'} at 127.0.0.1:50088
W20251214 20:52:24.465662 851 leader_election.cc:343] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 4b86f4adaf4540f48a4f42bc45e259e5
I20251214 20:52:24.466677 1977 tablet_replica.cc:333] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:24.467437 1202 ts_tablet_manager.cc:1403] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.495s user 0.010s sys 0.019s
I20251214 20:52:24.474680 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f891db00595b49ca9e6ae64721e4d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.476183 851 leader_election.cc:304] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:24.476135 876 tablet_service.cc:1505] Processing CreateTablet for tablet 71b8588773c442fa9add357455fc9d7d (DEFAULT_TABLE table=default.Table_4278bf82042b47c2b4d0e799f115c52f [id=2c037ca94c8a4c908d6354f2aa802bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.477033 2287 raft_consensus.cc:2749] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:24.478461 712 leader_election.cc:304] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:24.478829 2155 raft_consensus.cc:2804] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:24.478905 2155 raft_consensus.cc:493] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:24.478976 2155 raft_consensus.cc:3060] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:24.480806 1977 raft_consensus.cc:2243] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:24.481024 713 leader_election.cc:343] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 762068d560ff47918dd9fd8791083433
I20251214 20:52:24.481496 2286 consensus_queue.cc:1048] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251214 20:52:24.485225 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b8588773c442fa9add357455fc9d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.485433 878 tablet_service.cc:1505] Processing CreateTablet for tablet 489a43e7a1da41f4b1a8e80198eec212 (DEFAULT_TABLE table=default.Table_4cb054a62c404f1a952f39810b6d12f8 [id=515eec002ab94ec396c6bc70dae4d996]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.486014 1977 raft_consensus.cc:2272] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.487746 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_a4359e4ce4cb4a5caafc148f1dadef64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:24.416124 1024 tablet_service.cc:1505] Processing CreateTablet for tablet fa141b2570514c6aacb2850e4bfd1869 (DEFAULT_TABLE table=default.Table_47baf32513e6487fbd4ffde9bcba97ca [id=5ea1e541e5454f1ca96338573fda5705]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:24.488528 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4359e4ce4cb4a5caafc148f1dadef64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:24.488854 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa141b2570514c6aacb2850e4bfd1869. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.489017 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a43ee234b184275a6ab76b82fe5d283. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.416318 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 5ec7824f5eff4cc59c45acd3b3b3d194 (DEFAULT_TABLE table=default.Table_1e4dafc4ff8940ea90956623c915e3fd [id=be1be47d388347e3a6b7a80105688be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.491407 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec7824f5eff4cc59c45acd3b3b3d194. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.416462 1020 tablet_service.cc:1505] Processing CreateTablet for tablet 15b3216f3d2b443e8ab2f994445956f9 (DEFAULT_TABLE table=default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.492170 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15b3216f3d2b443e8ab2f994445956f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.493399 751 tablet_service.cc:1505] Processing CreateTablet for tablet ba548574d8784701974e611d7a9ad9b1 (DEFAULT_TABLE table=default.Table_0c936131512c4d99b3999d74fabf18d3 [id=1c5148c58b1c4fe39bac0697004ca484]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.416601 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 4f358df4d9414dcdb0d197f434e28118 (DEFAULT_TABLE table=default.Table_d8bb40c2324c4c0eb7e52aa96e920813 [id=3b04d86591b14695a334b68b0572f371]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.495570 1982 ts_tablet_manager.cc:1929] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:24.495687 1982 log.cc:1199] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/2a7fc15882e2474bb6885070be13b1fa
I20251214 20:52:24.416728 1022 tablet_service.cc:1505] Processing CreateTablet for tablet c3453fc996fb4d2b8cf0f80152098c43 (DEFAULT_TABLE table=default.Table_9196cd658f9041269835197751563391 [id=2f57c28c9c624a6ca1986c8831b85169]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.416990 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 2398f176ee044c15841eaa40cb74974c (DEFAULT_TABLE table=default.Table_32323ff67ecd4fe38fa02169904b88bf [id=b1ee89b437fb42a4b5e2c3d386ee0284]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.496004 1982 ts_tablet_manager.cc:1950] T 2a7fc15882e2474bb6885070be13b1fa P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:24.417018 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 5e92f54daf4f495fb6f026c179852362 (DEFAULT_TABLE table=default.Table_d6683a16a95f4274b82ff74dc7e73597 [id=573bdb65191b4f09b43190844f2b0008]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.417368 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 2593f67c7c6441b694d99286e3ba5883 (DEFAULT_TABLE table=default.Table_bde8bb1617d3450f999e9a7960d96641 [id=3a3dca8b90264d949ac9bead0b0743fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.417435 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 51366a6aefd54077877ec5d076394d79 (DEFAULT_TABLE table=default.Table_1191163db85341c5b41c4baf61bc8df5 [id=fcb196b8ebb14e1785ea355e7d406b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.417835 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 6583412e5a1849c8b93c87ba7fdc4647 (DEFAULT_TABLE table=default.Table_3474e4ca80964e3b8b7b66e1fde4815f [id=586a58c36e9a4d4494f4538ebcb9ce36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.418337 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 6302145a6e7b4297a58d9d7f3ad02dce (DEFAULT_TABLE table=default.Table_680d6597d5f0419e83bfca87ec6d7787 [id=54e0182cd3b9461aa593a11aa7709002]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.418846 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 71b8588773c442fa9add357455fc9d7d (DEFAULT_TABLE table=default.Table_4278bf82042b47c2b4d0e799f115c52f [id=2c037ca94c8a4c908d6354f2aa802bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.419927 1031 tablet_service.cc:1505] Processing CreateTablet for tablet f891db00595b49ca9e6ae64721e4d744 (DEFAULT_TABLE table=default.Table_19a53df5559b4eb5bfc7b41b54be081b [id=203681807a4b40b0a1f69387cfecff38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.420249 1023 tablet_service.cc:1505] Processing CreateTablet for tablet a6df388bdfbc40c183c4b14af2336a6a (DEFAULT_TABLE table=default.Table_0fb8f39742d245ac9d20fc7d6bf7e7c3 [id=552c8b213ba64d4bbdd3363cdc6625ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.420562 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 5a43ee234b184275a6ab76b82fe5d283 (DEFAULT_TABLE table=default.Table_0a325ff1f4634ce7ada593df8f5a9fd6 [id=88a6486d16184397acd4ef205ca46a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.427525 1018 tablet_service.cc:1505] Processing CreateTablet for tablet a88207c8cc5f4a6886de36c82c6b4c2f (DEFAULT_TABLE table=default.Table_bf73b75ea64f496da60f3a9a6afa5d6e [id=f41452b44a3c41db9752487fe11f201c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.437173 2184 consensus_queue.cc:1048] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:24.438347 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 489a43e7a1da41f4b1a8e80198eec212 (DEFAULT_TABLE table=default.Table_4cb054a62c404f1a952f39810b6d12f8 [id=515eec002ab94ec396c6bc70dae4d996]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.502818 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f891db00595b49ca9e6ae64721e4d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.503041 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a43ee234b184275a6ab76b82fe5d283. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.503484 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489a43e7a1da41f4b1a8e80198eec212. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.503870 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51366a6aefd54077877ec5d076394d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.504235 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6302145a6e7b4297a58d9d7f3ad02dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.504611 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e92f54daf4f495fb6f026c179852362. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.505308 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3453fc996fb4d2b8cf0f80152098c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.505694 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b8588773c442fa9add357455fc9d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.506076 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6583412e5a1849c8b93c87ba7fdc4647. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.506481 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6df388bdfbc40c183c4b14af2336a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.506894 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f358df4d9414dcdb0d197f434e28118. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.507273 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2398f176ee044c15841eaa40cb74974c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.510937 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2593f67c7c6441b694d99286e3ba5883. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.511463 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a88207c8cc5f4a6886de36c82c6b4c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.469100 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 727e7b61ae3942098712720d344c5310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:24.470492 1025 tablet_service.cc:1505] Processing CreateTablet for tablet f98be57ce1b14f20a3afcb79979b6c22 (DEFAULT_TABLE table=default.Table_da414f75240a40e6b20b644312217eae [id=0ff4524a6b97452fb0a8dce29d18a44a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:24.486865 989 consensus_peers.cc:597] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:24.583315 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 96469.4us
I20251214 20:52:24.586134 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98be57ce1b14f20a3afcb79979b6c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.585228 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f900ebe92efa476cac232131a818d449" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:24.591609 989 leader_election.cc:343] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.591778 1856 log.cc:1199] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/a063083f54ad49d2a842fa080d44ed85
I20251214 20:52:24.592151 1856 ts_tablet_manager.cc:1950] T a063083f54ad49d2a842fa080d44ed85 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:24.593037 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 171) took 3410 ms. Trace:
I20251214 20:52:24.593339 1856 rpcz_store.cc:276] 1214 20:52:21.182353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.182457 (+ 104us) service_pool.cc:225] Handling call
1214 20:52:24.361352 (+3178895us) ts_tablet_manager.cc:1151] Deleting tablet a063083f54ad49d2a842fa080d44ed85
1214 20:52:24.361360 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.399439 (+ 38079us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.402086 (+ 2647us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.593016 (+190930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3142075}
I20251214 20:52:24.593816 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 172) took 3410 ms. Trace:
I20251214 20:52:24.593959 1856 rpcz_store.cc:276] 1214 20:52:21.183653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.283228 (+ 99575us) service_pool.cc:225] Handling call
1214 20:52:24.593714 (+3310486us) ts_tablet_manager.cc:1151] Deleting tablet 84a0237e4ee34f94821d8647a400575b
1214 20:52:24.593723 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.593799 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3306356}
I20251214 20:52:24.594300 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 173) took 3310 ms. Trace:
I20251214 20:52:24.594414 1856 rpcz_store.cc:276] 1214 20:52:21.283683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.283739 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:24.594230 (+3310491us) ts_tablet_manager.cc:1151] Deleting tablet cd88d1e95ee04ef596863e1ae3687de0
1214 20:52:24.594238 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.594286 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3306797}
I20251214 20:52:24.594755 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 174) took 3310 ms. Trace:
I20251214 20:52:24.594887 1856 rpcz_store.cc:276] 1214 20:52:21.284037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.284085 (+ 48us) service_pool.cc:225] Handling call
1214 20:52:24.594655 (+3310570us) ts_tablet_manager.cc:1151] Deleting tablet 47e813373fbd482298312ead4f037c01
1214 20:52:24.594663 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.594739 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3307162}
I20251214 20:52:24.595036 2155 raft_consensus.cc:515] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.595271 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 176) took 3310 ms. Trace:
I20251214 20:52:24.595331 1856 rpcz_store.cc:276] 1214 20:52:21.284522 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.284565 (+ 43us) service_pool.cc:225] Handling call
1214 20:52:24.595166 (+3310601us) ts_tablet_manager.cc:1151] Deleting tablet 69f6108792fa4c2a9744e3248c76251f
1214 20:52:24.595175 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.595256 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3307638}
I20251214 20:52:24.596047 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa141b2570514c6aacb2850e4bfd1869. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.596216 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 177) took 3311 ms. Trace:
I20251214 20:52:24.596275 1856 rpcz_store.cc:276] 1214 20:52:21.284823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.284856 (+ 33us) service_pool.cc:225] Handling call
1214 20:52:24.595492 (+3310636us) ts_tablet_manager.cc:1151] Deleting tablet ec85857f047e44d78835305ddcb70768
1214 20:52:24.595500 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.596197 (+ 697us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3307956}
I20251214 20:52:24.596551 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 178) took 3311 ms. Trace:
I20251214 20:52:24.596611 1856 rpcz_store.cc:276] 1214 20:52:21.284940 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.285047 (+ 107us) service_pool.cc:225] Handling call
1214 20:52:24.596446 (+3311399us) ts_tablet_manager.cc:1151] Deleting tablet b1f4ea72c6274569a98cab6afdc55d55
1214 20:52:24.596454 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.596535 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3308865}
I20251214 20:52:24.596871 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 179) took 3311 ms. Trace:
I20251214 20:52:24.596923 1856 rpcz_store.cc:276] 1214 20:52:21.285047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.285185 (+ 138us) service_pool.cc:225] Handling call
1214 20:52:24.596770 (+3311585us) ts_tablet_manager.cc:1151] Deleting tablet 3d949abef1cb41a9905482d81a98d2c1
1214 20:52:24.596778 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.596858 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3309147}
I20251214 20:52:24.597159 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 180) took 3312 ms. Trace:
I20251214 20:52:24.597213 1856 rpcz_store.cc:276] 1214 20:52:21.285141 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.285185 (+ 44us) service_pool.cc:225] Handling call
1214 20:52:24.597084 (+3311899us) ts_tablet_manager.cc:1151] Deleting tablet ba50e32f7e724868bb891045a939e362
1214 20:52:24.597092 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.597147 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3309416}
I20251214 20:52:24.597453 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 181) took 3261 ms. Trace:
I20251214 20:52:24.597506 1856 rpcz_store.cc:276] 1214 20:52:21.335604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.335705 (+ 101us) service_pool.cc:225] Handling call
1214 20:52:24.597367 (+3261662us) ts_tablet_manager.cc:1151] Deleting tablet 06381efe75284e649fd6d1a9903cfa9b
1214 20:52:24.597374 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.597441 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3261298}
I20251214 20:52:24.597688 1856 tablet_replica.cc:333] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:24.597905 1856 raft_consensus.cc:2243] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:24.598048 1856 raft_consensus.cc:2272] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.598596 1856 ts_tablet_manager.cc:1916] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:24.599251 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f358df4d9414dcdb0d197f434e28118. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.599786 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51366a6aefd54077877ec5d076394d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.603840 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 197d3d340f6e4cbea0d621907b8c76d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.604120 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec7824f5eff4cc59c45acd3b3b3d194. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.604349 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15b3216f3d2b443e8ab2f994445956f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.604514 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a88207c8cc5f4a6886de36c82c6b4c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.604838 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3453fc996fb4d2b8cf0f80152098c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.605834 743 tablet_service.cc:1505] Processing CreateTablet for tablet a6df388bdfbc40c183c4b14af2336a6a (DEFAULT_TABLE table=default.Table_0fb8f39742d245ac9d20fc7d6bf7e7c3 [id=552c8b213ba64d4bbdd3363cdc6625ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.606215 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6df388bdfbc40c183c4b14af2336a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.607316 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2398f176ee044c15841eaa40cb74974c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.608520 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f891db00595b49ca9e6ae64721e4d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.608741 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6302145a6e7b4297a58d9d7f3ad02dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.609468 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2593f67c7c6441b694d99286e3ba5883. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.610123 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6583412e5a1849c8b93c87ba7fdc4647. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.619848 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet a063083f54ad49d2a842fa080d44ed85 (table default.Table_15002dcbc05542cebdf323d46303384b [id=71344b71ec00402f8df0c119fd8704bc]) successfully deleted
W20251214 20:52:24.620487 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 84a0237e4ee34f94821d8647a400575b because the tablet was not found. No further retry: Not found: Tablet not found: 84a0237e4ee34f94821d8647a400575b
W20251214 20:52:24.621104 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet cd88d1e95ee04ef596863e1ae3687de0 because the tablet was not found. No further retry: Not found: Tablet not found: cd88d1e95ee04ef596863e1ae3687de0
W20251214 20:52:24.621718 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 47e813373fbd482298312ead4f037c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e813373fbd482298312ead4f037c01 already in progress: creating tablet
W20251214 20:52:24.622249 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 69f6108792fa4c2a9744e3248c76251f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f6108792fa4c2a9744e3248c76251f already in progress: creating tablet
I20251214 20:52:24.622682 991 leader_election.cc:304] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:24.623353 2167 raft_consensus.cc:2804] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:24.623598 2167 raft_consensus.cc:697] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:24.623926 2167 consensus_queue.cc:237] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.637770 2155 leader_election.cc:290] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:24.649295 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98be57ce1b14f20a3afcb79979b6c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.649523 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba548574d8784701974e611d7a9ad9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.649842 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b8588773c442fa9add357455fc9d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.650729 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 167) took 2468 ms. Trace:
I20251214 20:52:24.650833 1982 rpcz_store.cc:276] 1214 20:52:22.182319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.182555 (+ 236us) service_pool.cc:225] Handling call
1214 20:52:24.146119 (+1963564us) ts_tablet_manager.cc:1151] Deleting tablet 2a7fc15882e2474bb6885070be13b1fa
1214 20:52:24.146127 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.313378 (+167251us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.495451 (+182073us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.650681 (+155230us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1958421}
I20251214 20:52:24.651292 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 168) took 2467 ms. Trace:
I20251214 20:52:24.651373 1982 rpcz_store.cc:276] 1214 20:52:22.184102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.184167 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:24.651188 (+2467021us) ts_tablet_manager.cc:1151] Deleting tablet 9523d72fd91445539d67e1eac2a80afd
1214 20:52:24.651197 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.651271 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2463430}
I20251214 20:52:24.651623 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 169) took 2466 ms. Trace:
I20251214 20:52:24.651683 1982 rpcz_store.cc:276] 1214 20:52:22.185587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.185925 (+ 338us) service_pool.cc:225] Handling call
1214 20:52:24.651555 (+2465630us) ts_tablet_manager.cc:1151] Deleting tablet fe4086fc7a4f4db0b7541f06c9833aab
1214 20:52:24.651562 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.651610 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2463737}
I20251214 20:52:24.651886 1982 tablet_replica.cc:333] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:24.651901 741 tablet_service.cc:1505] Processing CreateTablet for tablet 5a43ee234b184275a6ab76b82fe5d283 (DEFAULT_TABLE table=default.Table_0a325ff1f4634ce7ada593df8f5a9fd6 [id=88a6486d16184397acd4ef205ca46a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.652057 1982 raft_consensus.cc:2243] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:24.652297 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a43ee234b184275a6ab76b82fe5d283. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.652400 1982 raft_consensus.cc:2272] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.652971 1982 ts_tablet_manager.cc:1916] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:24.655318 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e92f54daf4f495fb6f026c179852362. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.655676 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 2a7fc15882e2474bb6885070be13b1fa (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]) successfully deleted
W20251214 20:52:24.656127 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 9523d72fd91445539d67e1eac2a80afd because the tablet was not found. No further retry: Not found: Tablet not found: 9523d72fd91445539d67e1eac2a80afd
W20251214 20:52:24.656323 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet fe4086fc7a4f4db0b7541f06c9833aab because the tablet was not found. No further retry: Not found: Tablet not found: fe4086fc7a4f4db0b7541f06c9833aab
I20251214 20:52:24.657588 750 tablet_service.cc:1505] Processing CreateTablet for tablet 489a43e7a1da41f4b1a8e80198eec212 (DEFAULT_TABLE table=default.Table_4cb054a62c404f1a952f39810b6d12f8 [id=515eec002ab94ec396c6bc70dae4d996]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:24.657945 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489a43e7a1da41f4b1a8e80198eec212. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.670389 1982 ts_tablet_manager.cc:1929] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:24.670542 1982 log.cc:1199] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/7fcfcfe81fe24d369eea696efc61d21c
I20251214 20:52:24.670910 1982 ts_tablet_manager.cc:1950] T 7fcfcfe81fe24d369eea696efc61d21c P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:24.671223 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 170) took 2474 ms. Trace:
I20251214 20:52:24.671308 1982 rpcz_store.cc:276] 1214 20:52:22.197305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.197494 (+ 189us) service_pool.cc:225] Handling call
1214 20:52:24.651845 (+2454351us) ts_tablet_manager.cc:1151] Deleting tablet 7fcfcfe81fe24d369eea696efc61d21c
1214 20:52:24.651852 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.660470 (+ 8618us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.663059 (+ 2589us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.671209 (+ 8150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2434700}
I20251214 20:52:24.671656 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 172) took 2452 ms. Trace:
I20251214 20:52:24.671732 1982 rpcz_store.cc:276] 1214 20:52:22.218753 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.218843 (+ 90us) service_pool.cc:225] Handling call
1214 20:52:24.671562 (+2452719us) ts_tablet_manager.cc:1151] Deleting tablet 472e4a5990e64d02b1d52dc6e86d7190
1214 20:52:24.671570 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.671643 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2452132}
I20251214 20:52:24.671947 1982 tablet_replica.cc:333] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:24.672118 1982 raft_consensus.cc:2243] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:24.672449 1982 raft_consensus.cc:2272] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.672976 1982 ts_tablet_manager.cc:1916] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:24.675557 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 7fcfcfe81fe24d369eea696efc61d21c (table default.Table_caa5b0dabd9d4ea8a8b16c550407dc53 [id=4c909a51df124835bd806b8c6e53214e]) successfully deleted
W20251214 20:52:24.675855 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 472e4a5990e64d02b1d52dc6e86d7190 because the tablet was not found. No further retry: Not found: Tablet not found: 472e4a5990e64d02b1d52dc6e86d7190
I20251214 20:52:24.692464 1206 tablet_bootstrap.cc:654] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:24.692404 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ec85857f047e44d78835305ddcb70768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec85857f047e44d78835305ddcb70768 already in progress: creating tablet
W20251214 20:52:24.692903 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b1f4ea72c6274569a98cab6afdc55d55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1f4ea72c6274569a98cab6afdc55d55 already in progress: creating tablet
W20251214 20:52:24.693300 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3d949abef1cb41a9905482d81a98d2c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d949abef1cb41a9905482d81a98d2c1 already in progress: creating tablet
W20251214 20:52:24.693666 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ba50e32f7e724868bb891045a939e362 because the tablet was not found. No further retry: Not found: Tablet not found: ba50e32f7e724868bb891045a939e362
W20251214 20:52:24.694685 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 06381efe75284e649fd6d1a9903cfa9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06381efe75284e649fd6d1a9903cfa9b already in progress: creating tablet
W20251214 20:52:24.697662 713 consensus_peers.cc:597] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d89ee602c524c96a21c0a235af07481. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:24.697846 713 leader_election.cc:343] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 762068d560ff47918dd9fd8791083433
I20251214 20:52:24.698946 1963 consensus_queue.cc:1048] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:24.699906 1982 ts_tablet_manager.cc:1929] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:24.700058 1982 log.cc:1199] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/d6c0f445cd7e44769d79ad5566fd8d97
I20251214 20:52:24.700423 1982 ts_tablet_manager.cc:1950] T d6c0f445cd7e44769d79ad5566fd8d97 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:24.700719 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 171) took 2484 ms. Trace:
I20251214 20:52:24.700786 1982 rpcz_store.cc:276] 1214 20:52:22.216340 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.216428 (+ 88us) service_pool.cc:225] Handling call
1214 20:52:24.671907 (+2455479us) ts_tablet_manager.cc:1151] Deleting tablet d6c0f445cd7e44769d79ad5566fd8d97
1214 20:52:24.671914 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.685740 (+ 13826us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.699747 (+ 14007us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.700701 (+ 954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2452349}
I20251214 20:52:24.701363 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet d6c0f445cd7e44769d79ad5566fd8d97 (table default.Table_d863e40d2a9e46649f9dc97f33a616db [id=cd65155def8244e59f08dac4d695567d]) successfully deleted
I20251214 20:52:24.704444 1203 tablet_bootstrap.cc:492] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:24.704589 1203 ts_tablet_manager.cc:1403] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.450s user 0.003s sys 0.009s
I20251214 20:52:24.704947 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 175) took 2485 ms. Trace:
I20251214 20:52:24.705045 1982 rpcz_store.cc:276] 1214 20:52:22.219919 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.220011 (+ 92us) service_pool.cc:225] Handling call
1214 20:52:24.704815 (+2484804us) ts_tablet_manager.cc:1151] Deleting tablet f3a2143b5508445492b970d2588cf1cf
1214 20:52:24.704831 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.704924 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2484496}
I20251214 20:52:24.705374 1203 raft_consensus.cc:359] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.705544 1203 raft_consensus.cc:385] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251214 20:52:24.705541 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f3a2143b5508445492b970d2588cf1cf because the tablet was not found. No further retry: Not found: Tablet not found: f3a2143b5508445492b970d2588cf1cf
I20251214 20:52:24.705595 1203 raft_consensus.cc:740] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:24.705899 1203 consensus_queue.cc:260] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.706426 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 176) took 2486 ms. Trace:
I20251214 20:52:24.706516 1982 rpcz_store.cc:276] 1214 20:52:22.220037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.220362 (+ 325us) service_pool.cc:225] Handling call
1214 20:52:24.706335 (+2485973us) ts_tablet_manager.cc:1151] Deleting tablet d768c45bc02b4d108798efbf3f6aaf7d
1214 20:52:24.706346 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.706409 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2485822}
I20251214 20:52:24.706760 1203 ts_tablet_manager.cc:1434] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251214 20:52:24.707008 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet d768c45bc02b4d108798efbf3f6aaf7d because the tablet was not found. No further retry: Not found: Tablet not found: d768c45bc02b4d108798efbf3f6aaf7d
I20251214 20:52:24.707897 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 173) took 2488 ms. Trace:
I20251214 20:52:24.707983 1982 rpcz_store.cc:276] 1214 20:52:22.219721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.220496 (+ 775us) service_pool.cc:225] Handling call
1214 20:52:24.707821 (+2487325us) ts_tablet_manager.cc:1151] Deleting tablet b652a8a80246443491551e76e68d2a8b
1214 20:52:24.707830 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.707883 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2487140}
W20251214 20:52:24.708359 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b652a8a80246443491551e76e68d2a8b because the tablet was not found. No further retry: Not found: Tablet not found: b652a8a80246443491551e76e68d2a8b
I20251214 20:52:24.709030 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 177) took 2488 ms. Trace:
I20251214 20:52:24.709106 1982 rpcz_store.cc:276] 1214 20:52:22.220150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.220556 (+ 406us) service_pool.cc:225] Handling call
1214 20:52:24.708955 (+2488399us) ts_tablet_manager.cc:1151] Deleting tablet f0d661a1bb034053a4634ff537eafc7d
1214 20:52:24.708963 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.709015 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2488225}
W20251214 20:52:24.709517 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f0d661a1bb034053a4634ff537eafc7d because the tablet was not found. No further retry: Not found: Tablet not found: f0d661a1bb034053a4634ff537eafc7d
I20251214 20:52:24.709898 1977 ts_tablet_manager.cc:1916] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:24.710251 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 174) took 2490 ms. Trace:
I20251214 20:52:24.710354 1982 rpcz_store.cc:276] 1214 20:52:22.219838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.220715 (+ 877us) service_pool.cc:225] Handling call
1214 20:52:24.710129 (+2489414us) ts_tablet_manager.cc:1151] Deleting tablet b315744c8cd043c5a8bdceec73a7e28b
1214 20:52:24.710138 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.710234 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2489226}
W20251214 20:52:24.710736 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b315744c8cd043c5a8bdceec73a7e28b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b315744c8cd043c5a8bdceec73a7e28b already in progress: creating tablet
I20251214 20:52:24.711623 1982 tablet_replica.cc:333] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:24.712194 1982 raft_consensus.cc:2243] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:24.712708 1982 raft_consensus.cc:2272] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.713675 1982 ts_tablet_manager.cc:1916] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:24.721048 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 876d2a1f1d1c4f6882a5857b43568ee5 consensus state has no leader
I20251214 20:52:24.723615 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 54e5fff1fc84419ab3d3131cb3a12d92
I20251214 20:52:24.726716 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 54e5fff1fc84419ab3d3131cb3a12d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:24.726872 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_4907bb640d114586b98e6fa0e514cb5a" } modify_external_catalogs: true
I20251214 20:52:24.727072 1203 tablet_bootstrap.cc:492] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:24.727108 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_4907bb640d114586b98e6fa0e514cb5a" } modify_external_catalogs: true
I20251214 20:52:24.728791 1203 tablet_bootstrap.cc:654] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:24.731758 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 7d4626d02ca548f3bf5a1a9142600bff
I20251214 20:52:24.733153 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d4626d02ca548f3bf5a1a9142600bff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:24.736443 1982 ts_tablet_manager.cc:1929] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:24.736964 1982 log.cc:1199] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/bb698ff30fcc47d09000cc68d2c5225b
I20251214 20:52:24.737358 1982 ts_tablet_manager.cc:1950] T bb698ff30fcc47d09000cc68d2c5225b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
W20251214 20:52:24.742811 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 57141.6us
I20251214 20:52:24.745833 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 178) took 2525 ms. Trace:
I20251214 20:52:24.746064 1982 rpcz_store.cc:276] 1214 20:52:22.220235 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.220763 (+ 528us) service_pool.cc:225] Handling call
1214 20:52:24.711573 (+2490810us) ts_tablet_manager.cc:1151] Deleting tablet bb698ff30fcc47d09000cc68d2c5225b
1214 20:52:24.711582 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.720968 (+ 9386us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.736314 (+ 15346us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.745808 (+ 9494us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2490658}
I20251214 20:52:24.746783 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet bb698ff30fcc47d09000cc68d2c5225b (table default.Table_7c95904dd8b044e4ad472681d5a5a274 [id=8c96acfc06ea4f00996fa066d719e418]) successfully deleted
I20251214 20:52:24.747323 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 179) took 2526 ms. Trace:
I20251214 20:52:24.747447 1982 rpcz_store.cc:276] 1214 20:52:22.220381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.220941 (+ 560us) service_pool.cc:225] Handling call
1214 20:52:24.747210 (+2526269us) ts_tablet_manager.cc:1151] Deleting tablet 324379e3217442c3addcfe58060b9ad7
1214 20:52:24.747221 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.747306 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2526087}
W20251214 20:52:24.748049 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 324379e3217442c3addcfe58060b9ad7 because the tablet was not found. No further retry: Not found: Tablet not found: 324379e3217442c3addcfe58060b9ad7
I20251214 20:52:24.748441 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 180) took 2512 ms. Trace:
I20251214 20:52:24.748556 1982 rpcz_store.cc:276] 1214 20:52:22.236236 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.236297 (+ 61us) service_pool.cc:225] Handling call
1214 20:52:24.748354 (+2512057us) ts_tablet_manager.cc:1151] Deleting tablet 6bd6856b800247de896f16eee66f2827
1214 20:52:24.748364 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.748425 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2511808}
W20251214 20:52:24.749136 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 6bd6856b800247de896f16eee66f2827 because the tablet was not found. No further retry: Not found: Tablet not found: 6bd6856b800247de896f16eee66f2827
I20251214 20:52:24.749527 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 181) took 2513 ms. Trace:
I20251214 20:52:24.749632 1982 rpcz_store.cc:276] 1214 20:52:22.236391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.236666 (+ 275us) service_pool.cc:225] Handling call
1214 20:52:24.749453 (+2512787us) ts_tablet_manager.cc:1151] Deleting tablet bd8226b431c6456885e928880938969d
1214 20:52:24.749462 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.749512 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2512580}
I20251214 20:52:24.750710 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 54e5fff1fc84419ab3d3131cb3a12d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:24.755548 2297 raft_consensus.cc:493] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:24.755721 2297 raft_consensus.cc:515] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.756374 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d4626d02ca548f3bf5a1a9142600bff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:24.756407 2297 leader_election.cc:290] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:24.757303 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b315744c8cd043c5a8bdceec73a7e28b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:24.757745 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet bd8226b431c6456885e928880938969d because the tablet was not found. No further retry: Not found: Tablet not found: bd8226b431c6456885e928880938969d
W20251214 20:52:24.758442 989 leader_election.cc:343] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.759078 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 183) took 2522 ms. Trace:
I20251214 20:52:24.763424 1982 rpcz_store.cc:276] 1214 20:52:22.236724 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.236911 (+ 187us) service_pool.cc:225] Handling call
1214 20:52:24.758902 (+2521991us) ts_tablet_manager.cc:1151] Deleting tablet 8faa7fd7315f485c82aee38bcf4cd8fc
1214 20:52:24.758921 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.759049 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2521769}
W20251214 20:52:24.764698 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8faa7fd7315f485c82aee38bcf4cd8fc because the tablet was not found. No further retry: Not found: Tablet not found: 8faa7fd7315f485c82aee38bcf4cd8fc
I20251214 20:52:24.765429 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 185) took 2528 ms. Trace:
I20251214 20:52:24.765990 1982 rpcz_store.cc:276] 1214 20:52:22.237058 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.237155 (+ 97us) service_pool.cc:225] Handling call
1214 20:52:24.765261 (+2528106us) ts_tablet_manager.cc:1151] Deleting tablet 6b4f582d8165439d86d9228c146dc41a
1214 20:52:24.765282 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.765405 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2527913}
W20251214 20:52:24.766782 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 6b4f582d8165439d86d9228c146dc41a because the tablet was not found. No further retry: Not found: Tablet not found: 6b4f582d8165439d86d9228c146dc41a
I20251214 20:52:24.767320 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 184) took 2530 ms. Trace:
I20251214 20:52:24.767743 1982 rpcz_store.cc:276] 1214 20:52:22.236912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.237367 (+ 455us) service_pool.cc:225] Handling call
1214 20:52:24.767216 (+2529849us) ts_tablet_manager.cc:1151] Deleting tablet c4b22043d81b493fb12116422b1d3fb1
1214 20:52:24.767227 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.767301 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2529689}
W20251214 20:52:24.768471 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c4b22043d81b493fb12116422b1d3fb1 because the tablet was not found. No further retry: Not found: Tablet not found: c4b22043d81b493fb12116422b1d3fb1
I20251214 20:52:24.768931 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 182) took 2532 ms. Trace:
I20251214 20:52:24.769328 1982 rpcz_store.cc:276] 1214 20:52:22.236492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.239392 (+ 2900us) service_pool.cc:225] Handling call
1214 20:52:24.768838 (+2529446us) ts_tablet_manager.cc:1151] Deleting tablet c25832da5c1644329e18653000f3330e
1214 20:52:24.768848 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.768912 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2529187}
W20251214 20:52:24.769985 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c25832da5c1644329e18653000f3330e because the tablet was not found. No further retry: Not found: Tablet not found: c25832da5c1644329e18653000f3330e
I20251214 20:52:24.770490 1982 tablet_replica.cc:333] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:24.770991 1982 raft_consensus.cc:2243] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:24.771677 1982 raft_consensus.cc:2272] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.772836 1982 ts_tablet_manager.cc:1916] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:24.773033 2318 tablet.cc:1722] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:24.786968 1856 ts_tablet_manager.cc:1929] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:24.787307 1856 log.cc:1199] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/4d6e7306254044449f1127fb963ca5f0
I20251214 20:52:24.787739 1856 ts_tablet_manager.cc:1950] T 4d6e7306254044449f1127fb963ca5f0 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:24.788681 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 182) took 3451 ms. Trace:
I20251214 20:52:24.789026 2317 tablet.cc:1722] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:24.793018 1982 ts_tablet_manager.cc:1929] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:24.797660 1856 rpcz_store.cc:276] 1214 20:52:21.336939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.337005 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:24.597651 (+3260646us) ts_tablet_manager.cc:1151] Deleting tablet 4d6e7306254044449f1127fb963ca5f0
1214 20:52:24.597658 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.740680 (+143022us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.786849 (+ 46169us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.788658 (+ 1809us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3258193}
I20251214 20:52:24.798249 1982 log.cc:1199] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/a063083f54ad49d2a842fa080d44ed85
I20251214 20:52:24.798751 1982 ts_tablet_manager.cc:1950] T a063083f54ad49d2a842fa080d44ed85 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:24.798774 2286 raft_consensus.cc:493] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:24.799065 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6df388bdfbc40c183c4b14af2336a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.799203 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 186) took 2533 ms. Trace:
I20251214 20:52:24.799292 1982 rpcz_store.cc:276] 1214 20:52:22.265410 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.265503 (+ 93us) service_pool.cc:225] Handling call
1214 20:52:24.770423 (+2504920us) ts_tablet_manager.cc:1151] Deleting tablet a063083f54ad49d2a842fa080d44ed85
1214 20:52:24.770433 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.785358 (+ 14925us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.792849 (+ 7491us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:24.799182 (+ 6333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2504621}
I20251214 20:52:24.799990 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet a063083f54ad49d2a842fa080d44ed85 (table default.Table_15002dcbc05542cebdf323d46303384b [id=71344b71ec00402f8df0c119fd8704bc]) successfully deleted
I20251214 20:52:24.800563 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 187) took 2534 ms. Trace:
I20251214 20:52:24.800640 1982 rpcz_store.cc:276] 1214 20:52:22.265608 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.265674 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:24.800427 (+2534753us) ts_tablet_manager.cc:1151] Deleting tablet 63497e77baeb4d328be2421b5732a28a
1214 20:52:24.800439 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.800544 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2534415}
W20251214 20:52:24.801083 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 63497e77baeb4d328be2421b5732a28a because the tablet was not found. No further retry: Not found: Tablet not found: 63497e77baeb4d328be2421b5732a28a
I20251214 20:52:24.801417 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 188) took 2535 ms. Trace:
I20251214 20:52:24.801496 1982 rpcz_store.cc:276] 1214 20:52:22.266253 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.266298 (+ 45us) service_pool.cc:225] Handling call
1214 20:52:24.801333 (+2535035us) ts_tablet_manager.cc:1151] Deleting tablet 2da66ec83e9140f7a605f0c86abff468
1214 20:52:24.801343 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.801400 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2534897}
W20251214 20:52:24.801995 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2da66ec83e9140f7a605f0c86abff468 because the tablet was not found. No further retry: Not found: Tablet not found: 2da66ec83e9140f7a605f0c86abff468
I20251214 20:52:24.802316 2155 raft_consensus.cc:493] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:24.802804 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 183) took 3465 ms. Trace:
I20251214 20:52:24.802646 2155 raft_consensus.cc:515] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.803570 2155 leader_election.cc:290] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:24.803815 1856 rpcz_store.cc:276] 1214 20:52:21.337417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.337483 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:24.802641 (+3465158us) ts_tablet_manager.cc:1151] Deleting tablet f84e9dfc23d34eb0a379e8ac5a442e59
1214 20:52:24.802654 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.802785 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3462795}
I20251214 20:52:24.804250 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 4d6e7306254044449f1127fb963ca5f0 (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]) successfully deleted
I20251214 20:52:24.804155 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5da9deeedce0421c9aeaebd71ba11b1b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:24.804986 1202 raft_consensus.cc:359] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.805250 1202 raft_consensus.cc:385] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:24.805349 1202 raft_consensus.cc:740] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:24.805611 1202 consensus_queue.cc:260] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.806134 1202 ts_tablet_manager.cc:1434] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.338s user 0.001s sys 0.001s
I20251214 20:52:24.802331 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 189) took 2535 ms. Trace:
I20251214 20:52:24.809569 1982 rpcz_store.cc:276] 1214 20:52:22.266939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.266984 (+ 45us) service_pool.cc:225] Handling call
1214 20:52:24.802216 (+2535232us) ts_tablet_manager.cc:1151] Deleting tablet ca263a80702f4f64aa7c872c2be08207
1214 20:52:24.802224 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.802314 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2535080}
W20251214 20:52:24.810115 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ca263a80702f4f64aa7c872c2be08207 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca263a80702f4f64aa7c872c2be08207 already in progress: creating tablet
I20251214 20:52:24.810182 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 190) took 2543 ms. Trace:
I20251214 20:52:24.810257 1982 rpcz_store.cc:276] 1214 20:52:22.267151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.267214 (+ 63us) service_pool.cc:225] Handling call
1214 20:52:24.810058 (+2542844us) ts_tablet_manager.cc:1151] Deleting tablet 40a9a6192c744282ab5672cbe15fd299
1214 20:52:24.810072 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.810164 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2542601}
W20251214 20:52:24.810845 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 40a9a6192c744282ab5672cbe15fd299 because the tablet was not found. No further retry: Not found: Tablet not found: 40a9a6192c744282ab5672cbe15fd299
I20251214 20:52:24.811372 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 54e5fff1fc84419ab3d3131cb3a12d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:24.814226 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f84e9dfc23d34eb0a379e8ac5a442e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f84e9dfc23d34eb0a379e8ac5a442e59 already in progress: creating tablet
I20251214 20:52:24.823861 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "762068d560ff47918dd9fd8791083433" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:24.824107 909 raft_consensus.cc:3060] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:24.829605 2286 raft_consensus.cc:515] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.833782 2286 leader_election.cc:290] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:24.834600 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebb26a5ae7224e9ca8a513c7b5013618" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:24.836023 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 191) took 2565 ms. Trace:
I20251214 20:52:24.836148 1982 rpcz_store.cc:276] 1214 20:52:22.270563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.270629 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:24.810783 (+2540154us) ts_tablet_manager.cc:1151] Deleting tablet 8d9f74fb2c6244a6887e3c922262bbab
1214 20:52:24.810796 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.835977 (+ 25181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2539896}
W20251214 20:52:24.836687 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 8d9f74fb2c6244a6887e3c922262bbab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9f74fb2c6244a6887e3c922262bbab already in progress: creating tablet
I20251214 20:52:24.837388 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 192) took 2566 ms. Trace:
I20251214 20:52:24.838900 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e92f54daf4f495fb6f026c179852362. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.837843 1982 rpcz_store.cc:276] 1214 20:52:22.271059 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.271119 (+ 60us) service_pool.cc:225] Handling call
1214 20:52:24.836862 (+2565743us) ts_tablet_manager.cc:1151] Deleting tablet 47e813373fbd482298312ead4f037c01
1214 20:52:24.836927 (+ 65us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.837317 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2565495}
W20251214 20:52:24.841609 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 47e813373fbd482298312ead4f037c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e813373fbd482298312ead4f037c01 already in progress: creating tablet
W20251214 20:52:24.853811 989 consensus_peers.cc:597] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:24.856359 892 consensus_meta.cc:287] T 15b3216f3d2b443e8ab2f994445956f9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent flushing consensus metadata: real 0.523s user 0.003s sys 0.000s
I20251214 20:52:24.858485 1202 tablet_bootstrap.cc:492] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:24.856375 909 raft_consensus.cc:2468] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:24.862984 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a88207c8cc5f4a6886de36c82c6b4c2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:24.863111 848 leader_election.cc:343] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.864203 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d4626d02ca548f3bf5a1a9142600bff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.868497 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6583412e5a1849c8b93c87ba7fdc4647. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.869865 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebb26a5ae7224e9ca8a513c7b5013618" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:24.870056 1052 raft_consensus.cc:2468] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:24.870978 1202 tablet_bootstrap.cc:654] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:24.872093 712 leader_election.cc:304] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:24.872736 2155 raft_consensus.cc:2804] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:24.872931 2155 raft_consensus.cc:697] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:24.873241 2155 consensus_queue.cc:237] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.881855 909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f900ebe92efa476cac232131a818d449" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:24.882108 909 raft_consensus.cc:2468] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:24.901013 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5da9deeedce0421c9aeaebd71ba11b1b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:24.902845 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6302145a6e7b4297a58d9d7f3ad02dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.904006 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489a43e7a1da41f4b1a8e80198eec212. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:24.905997 2167 consensus_queue.cc:1048] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:24.906709 991 leader_election.cc:304] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:24.907202 2167 raft_consensus.cc:2804] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:24.907274 2167 raft_consensus.cc:493] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:24.907331 2167 raft_consensus.cc:3060] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:24.909963 2167 raft_consensus.cc:515] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.910832 2167 leader_election.cc:290] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:24.911049 851 leader_election.cc:304] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:24.911084 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f900ebe92efa476cac232131a818d449" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:24.911401 2327 raft_consensus.cc:2804] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:24.911478 2327 raft_consensus.cc:493] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:24.911533 2327 raft_consensus.cc:3060] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:24.913791 2327 raft_consensus.cc:515] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.914381 2327 leader_election.cc:290] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:24.914601 989 leader_election.cc:343] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.915269 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebb26a5ae7224e9ca8a513c7b5013618" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:24.915422 712 leader_election.cc:304] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:24.916033 848 leader_election.cc:343] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.916235 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ebb26a5ae7224e9ca8a513c7b5013618" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:24.916399 1052 raft_consensus.cc:3060] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:24.918787 1052 raft_consensus.cc:2468] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:24.919924 851 leader_election.cc:304] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:24.920226 2327 raft_consensus.cc:2804] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:24.920464 769 raft_consensus.cc:3060] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:24.920619 2327 raft_consensus.cc:697] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:24.920890 2327 consensus_queue.cc:237] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.922518 2167 consensus_queue.cc:1048] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:24.924492 2167 consensus_queue.cc:1048] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:24.934545 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 185) took 3573 ms. Trace:
I20251214 20:52:24.939409 1856 rpcz_store.cc:276] 1214 20:52:21.361631 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.361780 (+ 149us) service_pool.cc:225] Handling call
1214 20:52:24.804665 (+3442885us) ts_tablet_manager.cc:1151] Deleting tablet 649f5bd67cbb4f61b7066269f0bf9266
1214 20:52:24.804678 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.934516 (+129838us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3442130}
W20251214 20:52:24.940256 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 649f5bd67cbb4f61b7066269f0bf9266 because the tablet was not found. No further retry: Not found: Tablet not found: 649f5bd67cbb4f61b7066269f0bf9266
I20251214 20:52:24.940485 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 187) took 3564 ms. Trace:
I20251214 20:52:24.940570 1856 rpcz_store.cc:276] 1214 20:52:21.376371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:21.376585 (+ 214us) service_pool.cc:225] Handling call
1214 20:52:24.940324 (+3563739us) ts_tablet_manager.cc:1151] Deleting tablet 9b1a937150ce4ecfa210c8465d682496
1214 20:52:24.940336 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.940467 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3558866}
I20251214 20:52:24.945536 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 201) took 2944 ms. Trace:
I20251214 20:52:24.945699 1856 rpcz_store.cc:276] 1214 20:52:22.001225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.001310 (+ 85us) service_pool.cc:225] Handling call
1214 20:52:24.945379 (+2944069us) ts_tablet_manager.cc:1151] Deleting tablet 2d164747cb294bd4b126b4420231867d
1214 20:52:24.945390 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.945518 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2943754}
I20251214 20:52:24.946235 1856 tablet_replica.cc:333] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:24.946477 1856 raft_consensus.cc:2243] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
W20251214 20:52:24.946483 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 9b1a937150ce4ecfa210c8465d682496 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b1a937150ce4ecfa210c8465d682496 already in progress: creating tablet
W20251214 20:52:24.946931 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2d164747cb294bd4b126b4420231867d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d164747cb294bd4b126b4420231867d already in progress: creating tablet
I20251214 20:52:24.946936 1856 raft_consensus.cc:2272] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:24.947607 1856 ts_tablet_manager.cc:1916] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:24.947937 2155 raft_consensus.cc:2749] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:24.950505 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 57f2314d33fa474ca36dafe0a0eae5ae
I20251214 20:52:24.951056 447 catalog_manager.cc:6724] ReplaceTablet: tablet 57f2314d33fa474ca36dafe0a0eae5ae deleted and replaced by tablet 1c82068013e8471c9309bcf8e8604ece
I20251214 20:52:24.954774 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f2314d33fa474ca36dafe0a0eae5ae with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82068013e8471c9309bcf8e8604ece) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:24.957185 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 193) took 2684 ms. Trace:
I20251214 20:52:24.957338 1982 rpcz_store.cc:276] 1214 20:52:22.272932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.272998 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:24.842451 (+2569453us) ts_tablet_manager.cc:1151] Deleting tablet 06381efe75284e649fd6d1a9903cfa9b
1214 20:52:24.842594 (+ 143us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.957165 (+114571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2569056}
W20251214 20:52:24.957962 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 06381efe75284e649fd6d1a9903cfa9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06381efe75284e649fd6d1a9903cfa9b already in progress: creating tablet
I20251214 20:52:24.960693 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f2314d33fa474ca36dafe0a0eae5ae with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82068013e8471c9309bcf8e8604ece) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:24.965072 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_0870823e1f29409f9d61ebee0bbd060d" } new_table_name: "default.Table_6a7108611108447db74a6feddaeea2b5" modify_external_catalogs: true
I20251214 20:52:24.965552 2286 raft_consensus.cc:493] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:24.966500 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_121c97f6b224464087b06c6fd31d4fa4" } modify_external_catalogs: true
I20251214 20:52:24.966642 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_121c97f6b224464087b06c6fd31d4fa4" } modify_external_catalogs: true
I20251214 20:52:24.965798 2286 raft_consensus.cc:515] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:24.968747 1650 raft_consensus.cc:493] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:24.968993 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5577714724264f51932f490a337aa8d2" } new_table_name: "default.Table_ad0accc4d4ef4dbc82e826110493c26a" modify_external_catalogs: true
I20251214 20:52:24.970567 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a91064ac3fb94282a02fbc867fdd581c" } new_table_name: "default.Table_913ff42ae53c475ab302d4e056e7eb48" modify_external_catalogs: true
I20251214 20:52:24.971686 452 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3369f89218b140e7a52da8221726f4e1" } modify_external_catalogs: true
I20251214 20:52:24.971858 452 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3369f89218b140e7a52da8221726f4e1" } modify_external_catalogs: true
I20251214 20:52:24.973418 449 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:24.974160 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_1de37433cd074bd494bcaa2064ea1823" } new_table_name: "default.Table_93fe3196f8c94960a5c5ee9e1f3923f1" modify_external_catalogs: true
I20251214 20:52:24.988278 2286 leader_election.cc:290] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:24.990522 848 consensus_peers.cc:597] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:24.957916 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 194) took 2684 ms. Trace:
I20251214 20:52:24.991693 1982 rpcz_store.cc:276] 1214 20:52:22.273087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.273131 (+ 44us) service_pool.cc:225] Handling call
1214 20:52:24.957810 (+2684679us) ts_tablet_manager.cc:1151] Deleting tablet b1f4ea72c6274569a98cab6afdc55d55
1214 20:52:24.957820 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.957902 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2684485}
I20251214 20:52:24.959363 1206 tablet_bootstrap.cc:492] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:24.992349 1206 ts_tablet_manager.cc:1403] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.627s user 0.004s sys 0.003s
W20251214 20:52:24.992318 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b1f4ea72c6274569a98cab6afdc55d55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1f4ea72c6274569a98cab6afdc55d55 already in progress: creating tablet
I20251214 20:52:24.968384 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fef1d6add764fd28a8f4995b337ea71" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:24.993080 1206 raft_consensus.cc:359] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.993247 1206 raft_consensus.cc:385] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:24.993286 1206 raft_consensus.cc:740] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
W20251214 20:52:24.993304 848 leader_election.cc:343] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.988481 1650 raft_consensus.cc:515] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:24.995589 1650 leader_election.cc:290] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:24.956753 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f2314d33fa474ca36dafe0a0eae5ae with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82068013e8471c9309bcf8e8604ece) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:24.996841 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "190afbb866854dd7b5393e04443752b7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:24.965083 991 leader_election.cc:343] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: b315744c8cd043c5a8bdceec73a7e28b
I20251214 20:52:24.997184 991 leader_election.cc:304] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:24.997471 848 leader_election.cc:343] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:24.989187 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fef1d6add764fd28a8f4995b337ea71" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:24.997813 2184 raft_consensus.cc:2749] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:24.996538 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "190afbb866854dd7b5393e04443752b7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:25.000460 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet ebb26a5ae7224e9ca8a513c7b5013618
I20251214 20:52:25.002619 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet ebb26a5ae7224e9ca8a513c7b5013618 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:25.006480 450 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:25.009500 742 tablet_service.cc:1552] Processing DeleteTablet for tablet ebb26a5ae7224e9ca8a513c7b5013618 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:25.014436 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 195) took 2740 ms. Trace:
I20251214 20:52:25.014581 1982 rpcz_store.cc:276] 1214 20:52:22.273480 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.273520 (+ 40us) service_pool.cc:225] Handling call
1214 20:52:25.014283 (+2740763us) ts_tablet_manager.cc:1151] Deleting tablet 3d949abef1cb41a9905482d81a98d2c1
1214 20:52:25.014296 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.014413 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2740606}
W20251214 20:52:25.015358 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3d949abef1cb41a9905482d81a98d2c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d949abef1cb41a9905482d81a98d2c1 already in progress: creating tablet
W20251214 20:52:25.016638 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 44800us
I20251214 20:52:25.022359 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 196) took 2748 ms. Trace:
I20251214 20:52:25.022588 1982 rpcz_store.cc:276] 1214 20:52:22.273760 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.273799 (+ 39us) service_pool.cc:225] Handling call
1214 20:52:25.022170 (+2748371us) ts_tablet_manager.cc:1151] Deleting tablet b2e83bca8ed642b6a1b705ef70fde27f
1214 20:52:25.022185 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.022317 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2748213}
W20251214 20:52:25.023192 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b2e83bca8ed642b6a1b705ef70fde27f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2e83bca8ed642b6a1b705ef70fde27f already in progress: creating tablet
I20251214 20:52:25.023662 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 197) took 2749 ms. Trace:
I20251214 20:52:25.023778 1982 rpcz_store.cc:276] 1214 20:52:22.274040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.274079 (+ 39us) service_pool.cc:225] Handling call
1214 20:52:25.023562 (+2749483us) ts_tablet_manager.cc:1151] Deleting tablet 292a28b12df246a8bc855d81c530746e
1214 20:52:25.023571 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.023646 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2749348}
W20251214 20:52:25.024425 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 292a28b12df246a8bc855d81c530746e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292a28b12df246a8bc855d81c530746e already in progress: creating tablet
I20251214 20:52:25.024824 1982 tablet_replica.cc:333] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
W20251214 20:52:25.024922 885 consensus_meta.cc:287] T f98be57ce1b14f20a3afcb79979b6c22 P a2e8fd1cb731446d88f8e510c7a52999: Time spent flushing consensus metadata: real 0.549s user 0.021s sys 0.000s
I20251214 20:52:25.025538 1982 raft_consensus.cc:2243] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:25.030639 851 leader_election.cc:343] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.030865 1203 tablet_bootstrap.cc:492] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:25.030854 851 leader_election.cc:304] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:25.031008 1203 ts_tablet_manager.cc:1403] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.304s user 0.005s sys 0.009s
W20251214 20:52:25.031334 851 leader_election.cc:343] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.031435 851 leader_election.cc:304] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:25.031738 1203 raft_consensus.cc:359] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.031983 2286 raft_consensus.cc:2749] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:25.032307 1203 raft_consensus.cc:385] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.032411 1203 raft_consensus.cc:740] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.032673 1203 consensus_queue.cc:260] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.034880 1977 ts_tablet_manager.cc:1929] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251214 20:52:25.040032 989 consensus_peers.cc:597] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:25.051836 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_7e2ed71ac05340a1be0ebd6af1b9dcc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:25.052603 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e2ed71ac05340a1be0ebd6af1b9dcc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:25.055955 453 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:25.057166 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_b58a6dfd93fb438684f7564325241f3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:25.057778 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b58a6dfd93fb438684f7564325241f3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:25.061123 989 consensus_peers.cc:597] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:25.062417 1203 ts_tablet_manager.cc:1434] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20251214 20:52:25.065222 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_09f08f5ffa814302a74b268170036a24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:25.065810 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09f08f5ffa814302a74b268170036a24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:25.075958 1203 tablet_bootstrap.cc:492] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:25.078145 1203 tablet_bootstrap.cc:654] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.036634 907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f900ebe92efa476cac232131a818d449" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:25.093792 907 raft_consensus.cc:3060] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:25.094937 1856 ts_tablet_manager.cc:1929] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:25.095032 1856 log.cc:1199] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/e16fe751f10f4616a00d0af811998729
I20251214 20:52:25.096527 907 raft_consensus.cc:2468] T f900ebe92efa476cac232131a818d449 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:25.047516 886 tablet_service.cc:1552] Processing DeleteTablet for tablet ebb26a5ae7224e9ca8a513c7b5013618 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:25.097213 991 leader_election.cc:304] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:25.100558 2167 raft_consensus.cc:2804] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:25.100683 2167 raft_consensus.cc:697] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:25.100932 2167 consensus_queue.cc:237] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.048794 1202 tablet_bootstrap.cc:492] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:25.050208 2196 raft_consensus.cc:2749] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:25.083669 1650 raft_consensus.cc:493] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.103756 1650 raft_consensus.cc:515] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.104388 1650 leader_election.cc:290] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:25.105733 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16816fa6b8c34207bb929147910b93cb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:25.105907 1050 raft_consensus.cc:2468] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:25.106554 851 leader_election.cc:304] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:25.106298 1202 ts_tablet_manager.cc:1403] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.275s user 0.006s sys 0.003s
I20251214 20:52:25.109249 1650 raft_consensus.cc:2804] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:25.109380 1650 raft_consensus.cc:493] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:25.109445 1650 raft_consensus.cc:3060] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:25.111004 2317 tablet.cc:1722] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:25.112758 1650 raft_consensus.cc:515] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.114117 1650 leader_election.cc:290] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:25.114734 1202 raft_consensus.cc:359] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.114948 1202 raft_consensus.cc:385] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.115038 1202 raft_consensus.cc:740] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.115351 1202 consensus_queue.cc:260] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.115976 1202 ts_tablet_manager.cc:1434] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:25.116536 1202 tablet_bootstrap.cc:492] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:25.117178 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16816fa6b8c34207bb929147910b93cb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:25.117367 1050 raft_consensus.cc:3060] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:25.118649 2184 consensus_queue.cc:1048] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:25.119098 1202 tablet_bootstrap.cc:654] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.119839 1050 raft_consensus.cc:2468] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:25.120666 851 leader_election.cc:304] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:25.123488 1650 raft_consensus.cc:2804] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:25.124236 1650 raft_consensus.cc:697] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:25.124523 1650 consensus_queue.cc:237] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.138338 2165 raft_consensus.cc:493] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.138585 2165 raft_consensus.cc:515] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.139323 1856 ts_tablet_manager.cc:1950] T e16fe751f10f4616a00d0af811998729 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:25.139384 2165 leader_election.cc:290] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:25.140856 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00cf456eda464f5ba22d6893f73b5cba" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:25.140998 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 202) took 3139 ms. Trace:
I20251214 20:52:25.141152 908 raft_consensus.cc:2468] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:25.141103 1856 rpcz_store.cc:276] 1214 20:52:22.001336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.001666 (+ 330us) service_pool.cc:225] Handling call
1214 20:52:24.946195 (+2944529us) ts_tablet_manager.cc:1151] Deleting tablet e16fe751f10f4616a00d0af811998729
1214 20:52:24.946202 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.001763 (+ 55561us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.094825 (+ 93062us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.140971 (+ 46146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2944338}
I20251214 20:52:25.141577 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet e16fe751f10f4616a00d0af811998729 (table default.Table_7923a9cba4ce4b27a12617e394ffbf2e [id=8fd9be1206a74aa1921e0dade8be3774]) successfully deleted
I20251214 20:52:25.141682 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 203) took 3140 ms. Trace:
I20251214 20:52:25.141737 1856 rpcz_store.cc:276] 1214 20:52:22.001444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.001885 (+ 441us) service_pool.cc:225] Handling call
1214 20:52:25.141535 (+3139650us) ts_tablet_manager.cc:1151] Deleting tablet 17550eb0ee6b476f8a8c304451f678b2
1214 20:52:25.141545 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.141667 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3139525}
I20251214 20:52:25.141834 991 leader_election.cc:304] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:25.142099 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 17550eb0ee6b476f8a8c304451f678b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17550eb0ee6b476f8a8c304451f678b2 already in progress: creating tablet
I20251214 20:52:25.142138 2165 raft_consensus.cc:2804] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:25.142168 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 204) took 3140 ms. Trace:
I20251214 20:52:25.142225 2165 raft_consensus.cc:493] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:25.142326 2165 raft_consensus.cc:3060] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:25.143203 1977 log.cc:1199] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/012b81223d0f4ae9b742014e329f11a9
I20251214 20:52:25.143731 1977 ts_tablet_manager.cc:1950] T 012b81223d0f4ae9b742014e329f11a9 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:25.144078 1982 raft_consensus.cc:2272] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:25.105023 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16816fa6b8c34207bb929147910b93cb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:25.114197 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16816fa6b8c34207bb929147910b93cb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:25.140497 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00cf456eda464f5ba22d6893f73b5cba" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:25.146014 989 leader_election.cc:343] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.145542 2165 raft_consensus.cc:515] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.147919 1982 ts_tablet_manager.cc:1916] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:25.148267 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00cf456eda464f5ba22d6893f73b5cba" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:25.148700 989 leader_election.cc:343] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.148901 2317 tablet.cc:1722] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251214 20:52:25.026196 1206 consensus_queue.cc:260] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.149209 2165 leader_election.cc:290] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:25.149832 1206 ts_tablet_manager.cc:1434] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.157s user 0.002s sys 0.000s
I20251214 20:52:25.150352 1206 tablet_bootstrap.cc:492] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:25.153807 2364 raft_consensus.cc:493] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.154138 2364 raft_consensus.cc:515] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.155682 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1389716f5ce44f588c21a34c7a7d4c8" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:25.155819 2364 leader_election.cc:290] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:25.156273 989 leader_election.cc:343] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.156911 1203 tablet_bootstrap.cc:492] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:25.157076 1203 ts_tablet_manager.cc:1403] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.000s
I20251214 20:52:25.157807 1203 raft_consensus.cc:359] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.157980 1203 raft_consensus.cc:385] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.158041 1203 raft_consensus.cc:740] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.158286 1203 consensus_queue.cc:260] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.158751 1203 ts_tablet_manager.cc:1434] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:25.161445 1203 tablet_bootstrap.cc:492] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:25.142230 1856 rpcz_store.cc:276] 1214 20:52:22.001562 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.001794 (+ 232us) service_pool.cc:225] Handling call
1214 20:52:25.142039 (+3140245us) ts_tablet_manager.cc:1151] Deleting tablet 8eb1006b0ba440dcba2f9de04dca2702
1214 20:52:25.142047 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.142153 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3,"tablet-delete.queue_time_us":3140030}
W20251214 20:52:25.165926 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8eb1006b0ba440dcba2f9de04dca2702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8eb1006b0ba440dcba2f9de04dca2702 already in progress: creating tablet
I20251214 20:52:25.166031 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 200) took 3165 ms. Trace:
I20251214 20:52:25.166112 1856 rpcz_store.cc:276] 1214 20:52:22.001107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.001624 (+ 517us) service_pool.cc:225] Handling call
1214 20:52:25.165877 (+3164253us) ts_tablet_manager.cc:1151] Deleting tablet 0c973e0111804b8b869068bdcc5869bf
1214 20:52:25.165891 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.166012 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3163750}
W20251214 20:52:25.166479 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 0c973e0111804b8b869068bdcc5869bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c973e0111804b8b869068bdcc5869bf already in progress: creating tablet
I20251214 20:52:25.166531 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 205) took 3035 ms. Trace:
I20251214 20:52:25.166587 1856 rpcz_store.cc:276] 1214 20:52:22.131505 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.133529 (+ 2024us) service_pool.cc:225] Handling call
1214 20:52:25.166437 (+3032908us) ts_tablet_manager.cc:1151] Deleting tablet b1df0dad2fb040a5a12e0401e00c87db
1214 20:52:25.166445 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.166517 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3032664}
W20251214 20:52:25.166960 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b1df0dad2fb040a5a12e0401e00c87db because the tablet was not found. No further retry: Not found: Tablet not found: b1df0dad2fb040a5a12e0401e00c87db
I20251214 20:52:25.167006 1856 tablet_replica.cc:333] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:25.167232 1856 raft_consensus.cc:2243] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:25.167551 1856 pending_rounds.cc:70] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:25.167659 1856 pending_rounds.cc:77] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493750781812736 op_type: NO_OP noop_request { }
I20251214 20:52:25.167825 1856 raft_consensus.cc:2889] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:25.167964 1856 raft_consensus.cc:2272] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:25.169005 1856 ts_tablet_manager.cc:1916] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:25.172955 1206 tablet_bootstrap.cc:654] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:25.181665 448 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e185d107619414a8bcc15ad496547dc consensus state has no leader
W20251214 20:52:25.182821 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d535d8f71b44c81bbba6937e804d900 consensus state has no leader
I20251214 20:52:25.188846 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:25.189967 448 master_service.cc:954] ReplaceTablet: received request to replace tablet d1389716f5ce44f588c21a34c7a7d4c8 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:25.191679 2318 tablet.cc:1722] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:25.192652 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_691250c9129d453585ad50b32a1e340e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:25.193384 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_691250c9129d453585ad50b32a1e340e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:25.194991 1203 tablet_bootstrap.cc:654] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.195392 2317 tablet.cc:1722] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251214 20:52:25.145308 848 leader_election.cc:343] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:25.212735 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 67427.2us
W20251214 20:52:25.212985 848 leader_election.cc:343] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.147609 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00cf456eda464f5ba22d6893f73b5cba" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:25.213711 908 raft_consensus.cc:3060] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:25.203524 1202 tablet_bootstrap.cc:492] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:25.214500 1202 ts_tablet_manager.cc:1403] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.000s
I20251214 20:52:25.215451 1202 raft_consensus.cc:359] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.215654 1202 raft_consensus.cc:385] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.215704 1202 raft_consensus.cc:740] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.215960 1202 consensus_queue.cc:260] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.216480 1202 ts_tablet_manager.cc:1434] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:25.216671 908 raft_consensus.cc:2468] T 00cf456eda464f5ba22d6893f73b5cba P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:25.216832 1202 tablet_bootstrap.cc:492] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:25.222720 2327 consensus_queue.cc:1048] T 4408013cf49e46d0b18d24b48483ea49 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:25.223593 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 193) took 2231 ms. Trace:
I20251214 20:52:25.223733 1977 rpcz_store.cc:276] 1214 20:52:22.992627 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.992668 (+ 41us) service_pool.cc:225] Handling call
1214 20:52:24.424905 (+1432237us) ts_tablet_manager.cc:1151] Deleting tablet 012b81223d0f4ae9b742014e329f11a9
1214 20:52:24.454809 (+ 29904us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:24.926507 (+471698us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.034682 (+108175us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.223573 (+188891us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1432101}
I20251214 20:52:25.224323 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 012b81223d0f4ae9b742014e329f11a9 (table default.Table_b151b87a38944a7a89d0fe6ff10ae68b [id=7b72926c96404fa5b84cfb86e6ae10a7]) successfully deleted
I20251214 20:52:25.224802 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 194) took 2231 ms. Trace:
I20251214 20:52:25.224943 1977 rpcz_store.cc:276] 1214 20:52:22.992982 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.993021 (+ 39us) service_pool.cc:225] Handling call
1214 20:52:25.224677 (+2231656us) ts_tablet_manager.cc:1151] Deleting tablet 8d9f74fb2c6244a6887e3c922262bbab
1214 20:52:25.224688 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.224785 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2231520}
W20251214 20:52:25.225539 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8d9f74fb2c6244a6887e3c922262bbab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9f74fb2c6244a6887e3c922262bbab already in progress: creating tablet
I20251214 20:52:25.225956 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 195) took 2232 ms. Trace:
I20251214 20:52:25.226234 1977 rpcz_store.cc:276] 1214 20:52:22.993328 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.993364 (+ 36us) service_pool.cc:225] Handling call
1214 20:52:25.225833 (+2232469us) ts_tablet_manager.cc:1151] Deleting tablet 9b1a937150ce4ecfa210c8465d682496
1214 20:52:25.225843 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.225939 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232341}
W20251214 20:52:25.234556 441 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 1e185d107619414a8bcc15ad496547dc (table default.Table_ad0accc4d4ef4dbc82e826110493c26a [id=c3e26e52ceef4e29a1ae13efb3d380d4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1e185d107619414a8bcc15ad496547dc consensus state has no leader
I20251214 20:52:25.237843 2317 tablet.cc:1722] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:25.236801 991 leader_election.cc:304] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:25.242007 2368 raft_consensus.cc:2804] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:25.242211 2368 raft_consensus.cc:697] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:25.245736 1856 ts_tablet_manager.cc:1929] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:25.245987 1856 log.cc:1199] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/012b81223d0f4ae9b742014e329f11a9
I20251214 20:52:25.254369 2318 tablet.cc:1722] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:25.263055 2374 tablet.cc:1722] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251214 20:52:25.267460 1856 ts_tablet_manager.cc:1950] T 012b81223d0f4ae9b742014e329f11a9 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:25.268396 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 220) took 3135 ms. Trace:
I20251214 20:52:25.268546 1856 rpcz_store.cc:276] 1214 20:52:22.133163 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.133708 (+ 545us) service_pool.cc:225] Handling call
1214 20:52:25.166959 (+3033251us) ts_tablet_manager.cc:1151] Deleting tablet 012b81223d0f4ae9b742014e329f11a9
1214 20:52:25.166967 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.224625 (+ 57658us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.245615 (+ 20990us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.268376 (+ 22761us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3033083}
I20251214 20:52:25.269114 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 206) took 3137 ms. Trace:
I20251214 20:52:25.269181 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 012b81223d0f4ae9b742014e329f11a9 (table default.Table_b151b87a38944a7a89d0fe6ff10ae68b [id=7b72926c96404fa5b84cfb86e6ae10a7]) successfully deleted
I20251214 20:52:25.269264 1856 rpcz_store.cc:276] 1214 20:52:22.132017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.133820 (+ 1803us) service_pool.cc:225] Handling call
1214 20:52:25.268960 (+3135140us) ts_tablet_manager.cc:1151] Deleting tablet 98b7cac3930a4a9eb16ac413c8f1e092
1214 20:52:25.268971 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.269095 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3135022}
W20251214 20:52:25.269634 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 98b7cac3930a4a9eb16ac413c8f1e092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98b7cac3930a4a9eb16ac413c8f1e092 already in progress: creating tablet
I20251214 20:52:25.279577 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 221) took 3146 ms. Trace:
I20251214 20:52:25.279763 1856 rpcz_store.cc:276] 1214 20:52:22.133251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.133903 (+ 652us) service_pool.cc:225] Handling call
1214 20:52:25.269500 (+3135597us) ts_tablet_manager.cc:1151] Deleting tablet 569a02142c71427d91868b02695ae12b
1214 20:52:25.269509 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.279550 (+ 10041us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3135459}
W20251214 20:52:25.282109 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 569a02142c71427d91868b02695ae12b because the tablet was not found. No further retry: Not found: Tablet not found: 569a02142c71427d91868b02695ae12b
W20251214 20:52:25.282932 2237 kernel_stack_watchdog.cc:198] Thread 1769 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 292ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fba23081980 (unknown) at ??:0
@ 0x93e827 __sanitizer::internal_sched_yield() at ??:?
@ 0x8a3edd __interceptor_pthread_create at ??:?
@ 0x7fba1fa6a0f1 kudu::Thread::StartThread() at ??:0
@ 0xa76197 kudu::Thread::Create() at ??:?
@ 0x7fba26a06583 kudu::rpc::ServicePool::Init() at ??:0
@ 0x7fba28eaff51 kudu::RpcServer::RegisterService() at ??:0
@ 0x7fba28ed1368 kudu::server::ServerBase::RegisterService() at ??:0
@ 0x7fba336ffa5b kudu::master::Master::StartAsync() at ??:0
@ 0x7fba336ff017 kudu::master::Master::Start() at ??:0
@ 0x7fba33761a5b kudu::master::RunMasterServer() at ??:0
@ 0xaa23b7 kudu::tools::(anonymous namespace)::MasterRun() at ??:?
@ 0x9bd341 std::_Function_handler<>::_M_invoke() at ??:?
@ 0x7fba34109179 std::function<>::operator()() at ??:0
@ 0x7fba34104c95 kudu::tools::Action::Run() at ??:0
@ 0xbd1144 kudu::tools::DispatchCommand() at ??:?
I20251214 20:52:25.242480 2368 consensus_queue.cc:237] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.290495 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 207) took 3158 ms. Trace:
I20251214 20:52:25.290609 1856 rpcz_store.cc:276] 1214 20:52:22.132137 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.133965 (+ 1828us) service_pool.cc:225] Handling call
1214 20:52:25.280294 (+3146329us) ts_tablet_manager.cc:1151] Deleting tablet a10663bd306d40e7a6375d06d69be22d
1214 20:52:25.280307 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.290467 (+ 10160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3146214}
W20251214 20:52:25.294682 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 453cf734f8b34b59aa246d618ffeade5 consensus state has no leader
I20251214 20:52:25.304126 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 222) took 3170 ms. Trace:
I20251214 20:52:25.304227 1856 rpcz_store.cc:276] 1214 20:52:22.133318 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.134073 (+ 755us) service_pool.cc:225] Handling call
1214 20:52:25.290872 (+3156799us) ts_tablet_manager.cc:1151] Deleting tablet 2b48bb260e854beb9ec61ce9c79b46c2
1214 20:52:25.290886 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.304097 (+ 13211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3156679}
W20251214 20:52:25.305310 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a10663bd306d40e7a6375d06d69be22d because the tablet was not found. No further retry: Not found: Tablet not found: a10663bd306d40e7a6375d06d69be22d
W20251214 20:52:25.306401 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2b48bb260e854beb9ec61ce9c79b46c2 because the tablet was not found. No further retry: Not found: Tablet not found: 2b48bb260e854beb9ec61ce9c79b46c2
I20251214 20:52:25.314656 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 208) took 3182 ms. Trace:
I20251214 20:52:25.314764 1856 rpcz_store.cc:276] 1214 20:52:22.132219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.134097 (+ 1878us) service_pool.cc:225] Handling call
1214 20:52:25.304648 (+3170551us) ts_tablet_manager.cc:1151] Deleting tablet c9b7149a436a4cab80b39adccf8567f9
1214 20:52:25.304659 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.314630 (+ 9971us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3170419}
I20251214 20:52:25.315177 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 209) took 3182 ms. Trace:
I20251214 20:52:25.315249 1856 rpcz_store.cc:276] 1214 20:52:22.132308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.134252 (+ 1944us) service_pool.cc:225] Handling call
1214 20:52:25.315034 (+3180782us) ts_tablet_manager.cc:1151] Deleting tablet 4b86f4adaf4540f48a4f42bc45e259e5
1214 20:52:25.315054 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.315162 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3180639}
W20251214 20:52:25.315326 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c9b7149a436a4cab80b39adccf8567f9 because the tablet was not found. No further retry: Not found: Tablet not found: c9b7149a436a4cab80b39adccf8567f9
I20251214 20:52:25.315564 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 210) took 3183 ms. Trace:
I20251214 20:52:25.315631 1856 rpcz_store.cc:276] 1214 20:52:22.132389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.134412 (+ 2023us) service_pool.cc:225] Handling call
1214 20:52:25.315467 (+3181055us) ts_tablet_manager.cc:1151] Deleting tablet fe4086fc7a4f4db0b7541f06c9833aab
1214 20:52:25.315476 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.315551 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3180931}
I20251214 20:52:25.315896 1856 tablet_replica.cc:333] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:25.315773 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 4b86f4adaf4540f48a4f42bc45e259e5 because the tablet was not found. No further retry: Not found: Tablet not found: 4b86f4adaf4540f48a4f42bc45e259e5
W20251214 20:52:25.316111 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet fe4086fc7a4f4db0b7541f06c9833aab because the tablet was not found. No further retry: Not found: Tablet not found: fe4086fc7a4f4db0b7541f06c9833aab
I20251214 20:52:25.316102 1856 raft_consensus.cc:2243] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:25.316267 1856 raft_consensus.cc:2272] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:25.249203 1202 tablet_bootstrap.cc:654] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.320422 1856 ts_tablet_manager.cc:1916] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:25.320537 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 9b1a937150ce4ecfa210c8465d682496 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b1a937150ce4ecfa210c8465d682496 already in progress: creating tablet
I20251214 20:52:25.251380 2356 tablet.cc:1722] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:25.255520 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1389716f5ce44f588c21a34c7a7d4c8" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:25.257932 1650 raft_consensus.cc:493] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.262318 2373 tablet.cc:1722] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:25.262620 2375 tablet.cc:1722] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251214 20:52:25.274773 2376 tablet.cc:1722] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:25.325955 1650 raft_consensus.cc:515] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.327283 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe0c3c8a1d5b46cfacd35d3f0d82e432" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:25.327841 851 leader_election.cc:343] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.328758 1650 leader_election.cc:290] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:25.329141 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe0c3c8a1d5b46cfacd35d3f0d82e432" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:25.329730 848 leader_election.cc:343] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.329845 848 leader_election.cc:304] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:25.330267 1650 raft_consensus.cc:2749] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:25.330883 991 leader_election.cc:343] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.331055 991 leader_election.cc:304] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:25.331576 2364 raft_consensus.cc:2749] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:25.331808 1206 tablet_bootstrap.cc:492] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:25.331940 1206 ts_tablet_manager.cc:1403] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.182s user 0.002s sys 0.001s
I20251214 20:52:25.336803 1203 tablet_bootstrap.cc:492] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:25.336936 1203 ts_tablet_manager.cc:1403] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.176s user 0.012s sys 0.004s
I20251214 20:52:25.337725 1203 raft_consensus.cc:359] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.337884 1203 raft_consensus.cc:385] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.337930 1203 raft_consensus.cc:740] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.342828 1203 consensus_queue.cc:260] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.349313 452 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 248e3666fd13434e97378eb5a98dfadd
I20251214 20:52:25.350015 452 master_service.cc:954] ReplaceTablet: received request to replace tablet 1eff8a23d48c43e2bdbfb34f4bab6edf from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:25.350176 2375 tablet.cc:1722] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:25.352118 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 248e3666fd13434e97378eb5a98dfadd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:25.354000 1203 ts_tablet_manager.cc:1434] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20251214 20:52:25.355366 1203 tablet_bootstrap.cc:492] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:25.357300 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 248e3666fd13434e97378eb5a98dfadd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:25.360355 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 248e3666fd13434e97378eb5a98dfadd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:25.320955 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 196) took 2327 ms. Trace:
I20251214 20:52:25.366133 1977 rpcz_store.cc:276] 1214 20:52:22.993664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.993704 (+ 40us) service_pool.cc:225] Handling call
1214 20:52:25.320810 (+2327106us) ts_tablet_manager.cc:1151] Deleting tablet 06381efe75284e649fd6d1a9903cfa9b
1214 20:52:25.320824 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.320932 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2326964}
W20251214 20:52:25.367048 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 06381efe75284e649fd6d1a9903cfa9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06381efe75284e649fd6d1a9903cfa9b already in progress: creating tablet
I20251214 20:52:25.368095 1203 tablet_bootstrap.cc:654] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.368402 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet e6ebfc373fbf41b1920c271a15302484
I20251214 20:52:25.369113 455 master_service.cc:954] ReplaceTablet: received request to replace tablet a142157acbcd44adb6099b4428103ee5 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:25.370532 886 tablet_service.cc:1552] Processing DeleteTablet for tablet e6ebfc373fbf41b1920c271a15302484 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:25.371181 744 tablet_service.cc:1552] Processing DeleteTablet for tablet e6ebfc373fbf41b1920c271a15302484 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:25.371253 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet e6ebfc373fbf41b1920c271a15302484 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:25.375568 1977 tablet_replica.cc:333] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:25.376240 1856 ts_tablet_manager.cc:1929] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:25.376353 1856 log.cc:1199] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/bf8643c7e4094f529cd55656c7b3edb6
I20251214 20:52:25.376672 1856 ts_tablet_manager.cc:1950] T bf8643c7e4094f529cd55656c7b3edb6 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:25.379765 1206 raft_consensus.cc:359] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.379988 1206 raft_consensus.cc:385] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.380044 1206 raft_consensus.cc:740] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.380262 1206 consensus_queue.cc:260] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.380893 1206 ts_tablet_manager.cc:1434] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.049s user 0.003s sys 0.000s
I20251214 20:52:25.381549 1206 tablet_bootstrap.cc:492] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:25.388294 1203 tablet_bootstrap.cc:492] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:25.388440 1203 ts_tablet_manager.cc:1403] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20251214 20:52:25.389276 1203 raft_consensus.cc:359] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.389448 1203 raft_consensus.cc:385] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.389497 1203 raft_consensus.cc:740] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.389734 1203 consensus_queue.cc:260] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.391366 1203 ts_tablet_manager.cc:1434] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:25.392530 1203 tablet_bootstrap.cc:492] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:25.394230 1203 tablet_bootstrap.cc:654] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.395037 1977 raft_consensus.cc:2243] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:25.396149 1977 raft_consensus.cc:2272] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:25.400028 1977 ts_tablet_manager.cc:1916] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:25.403231 1769 thread.cc:641] rpc worker (service pool kudu.consensus.ConsensusService) Time spent creating pthread: real 0.535s user 0.184s sys 0.211s
W20251214 20:52:25.403409 1769 thread.cc:608] rpc worker (service pool kudu.consensus.ConsensusService) Time spent starting thread: real 0.535s user 0.184s sys 0.211s
I20251214 20:52:25.411572 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_affc01186d6d429bb07259ca92056991" } new_table_name: "default.Table_68b81e8dee2b457ea98b839dca8e5464" modify_external_catalogs: true
I20251214 20:52:25.417709 1202 tablet_bootstrap.cc:492] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:25.417919 1202 ts_tablet_manager.cc:1403] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.201s user 0.006s sys 0.007s
I20251214 20:52:25.418459 1203 tablet_bootstrap.cc:492] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:25.418612 1203 ts_tablet_manager.cc:1403] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20251214 20:52:25.418862 453 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_ed823cd2c6d94ed2b4707f427c8a2867 [id=0d173bfba2b343dbac6a02a5ff84c828] alter complete (version 2)
I20251214 20:52:25.418895 1202 raft_consensus.cc:359] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.419106 1202 raft_consensus.cc:385] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.419191 1202 raft_consensus.cc:740] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.419472 1203 raft_consensus.cc:359] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.419643 1203 raft_consensus.cc:385] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.419697 1203 raft_consensus.cc:740] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.419579 1202 consensus_queue.cc:260] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.420660 1203 consensus_queue.cc:260] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.422431 1202 ts_tablet_manager.cc:1434] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:25.422703 1202 tablet_bootstrap.cc:492] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:25.428800 1982 ts_tablet_manager.cc:1929] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:25.429348 1203 ts_tablet_manager.cc:1434] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251214 20:52:25.430521 1203 tablet_bootstrap.cc:492] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:25.432044 1203 tablet_bootstrap.cc:654] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.437431 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ec9d2344b1bc430784fe937726619f77" } modify_external_catalogs: true
I20251214 20:52:25.437673 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ec9d2344b1bc430784fe937726619f77" } modify_external_catalogs: true
I20251214 20:52:25.438577 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8f5893ce749b45e0889fb1735f5a59ba" } modify_external_catalogs: true
I20251214 20:52:25.438733 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8f5893ce749b45e0889fb1735f5a59ba" } modify_external_catalogs: true
I20251214 20:52:25.440644 1202 tablet_bootstrap.cc:654] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.442761 1203 tablet_bootstrap.cc:492] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:25.442909 1203 ts_tablet_manager.cc:1403] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20251214 20:52:25.443704 1203 raft_consensus.cc:359] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.443872 1203 raft_consensus.cc:385] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.443920 1203 raft_consensus.cc:740] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.444190 1203 consensus_queue.cc:260] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.450223 1203 ts_tablet_manager.cc:1434] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:25.451195 1982 log.cc:1199] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ee7d709316b24b8eb223a838710c2979
I20251214 20:52:25.451480 1203 tablet_bootstrap.cc:492] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:25.451647 1982 ts_tablet_manager.cc:1950] T ee7d709316b24b8eb223a838710c2979 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:25.451990 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 198) took 3177 ms. Trace:
I20251214 20:52:25.452067 1982 rpcz_store.cc:276] 1214 20:52:22.274382 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.274598 (+ 216us) service_pool.cc:225] Handling call
1214 20:52:25.024780 (+2750182us) ts_tablet_manager.cc:1151] Deleting tablet ee7d709316b24b8eb223a838710c2979
1214 20:52:25.024788 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.286471 (+261683us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.428554 (+142083us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.451971 (+ 23417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2750022}
I20251214 20:52:25.452628 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ee7d709316b24b8eb223a838710c2979 (table default.Table_aa507956e1b54af68730c20f7c10e842 [id=f249932331aa4016806e482e8c7c4179]) successfully deleted
I20251214 20:52:25.452970 1203 tablet_bootstrap.cc:654] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.453089 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 199) took 3178 ms. Trace:
I20251214 20:52:25.453161 1982 rpcz_store.cc:276] 1214 20:52:22.274620 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.274681 (+ 61us) service_pool.cc:225] Handling call
1214 20:52:25.452951 (+3178270us) ts_tablet_manager.cc:1151] Deleting tablet 7dd657f3baff40bb8cb7f4648367f7a1
1214 20:52:25.452962 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.453071 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3178066}
W20251214 20:52:25.453612 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 7dd657f3baff40bb8cb7f4648367f7a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dd657f3baff40bb8cb7f4648367f7a1 already in progress: creating tablet
I20251214 20:52:25.453963 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 200) took 3153 ms. Trace:
I20251214 20:52:25.454037 1982 rpcz_store.cc:276] 1214 20:52:22.300111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.300211 (+ 100us) service_pool.cc:225] Handling call
1214 20:52:25.453859 (+3153648us) ts_tablet_manager.cc:1151] Deleting tablet dfe50a6219134431b0a025317932609b
1214 20:52:25.453868 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.453947 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3150403}
I20251214 20:52:25.454319 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 212) took 3321 ms. Trace:
W20251214 20:52:25.454461 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet dfe50a6219134431b0a025317932609b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfe50a6219134431b0a025317932609b already in progress: creating tablet
I20251214 20:52:25.454396 1856 rpcz_store.cc:276] 1214 20:52:22.132555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.134764 (+ 2209us) service_pool.cc:225] Handling call
1214 20:52:25.315847 (+3181083us) ts_tablet_manager.cc:1151] Deleting tablet bf8643c7e4094f529cd55656c7b3edb6
1214 20:52:25.315856 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.332907 (+ 17051us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.376112 (+ 43205us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.454299 (+ 78187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3180928}
I20251214 20:52:25.454824 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet bf8643c7e4094f529cd55656c7b3edb6 (table default.Table_1bba0cb42a0c4aa986d55fdc492ffd46 [id=ce55ad7e80fe40e99895c52abb55e986]) successfully deleted
I20251214 20:52:25.454945 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 213) took 3322 ms. Trace:
I20251214 20:52:25.455030 1856 rpcz_store.cc:276] 1214 20:52:22.132633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.134939 (+ 2306us) service_pool.cc:225] Handling call
1214 20:52:25.454850 (+3319911us) ts_tablet_manager.cc:1151] Deleting tablet a0279992d1f2419ab1caae116a0391b2
1214 20:52:25.454861 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.454930 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3319787}
I20251214 20:52:25.455193 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 202) took 3126 ms. Trace:
W20251214 20:52:25.455449 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a0279992d1f2419ab1caae116a0391b2 because the tablet was not found. No further retry: Not found: Tablet not found: a0279992d1f2419ab1caae116a0391b2
I20251214 20:52:25.455525 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 214) took 3322 ms. Trace:
I20251214 20:52:25.455587 1856 rpcz_store.cc:276] 1214 20:52:22.132701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.135077 (+ 2376us) service_pool.cc:225] Handling call
1214 20:52:25.455417 (+3320340us) ts_tablet_manager.cc:1151] Deleting tablet 8d9f74fb2c6244a6887e3c922262bbab
1214 20:52:25.455427 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.455509 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3320187}
I20251214 20:52:25.455255 1982 rpcz_store.cc:276] 1214 20:52:22.328604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.328670 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:25.455058 (+3126388us) ts_tablet_manager.cc:1151] Deleting tablet 242a77ae57174de48f4eaf43384ee0ba
1214 20:52:25.455068 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.455178 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3126169}
W20251214 20:52:25.455947 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 242a77ae57174de48f4eaf43384ee0ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 242a77ae57174de48f4eaf43384ee0ba already in progress: creating tablet
I20251214 20:52:25.455968 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 215) took 3323 ms. Trace:
I20251214 20:52:25.456027 1856 rpcz_store.cc:276] 1214 20:52:22.132789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.135248 (+ 2459us) service_pool.cc:225] Handling call
1214 20:52:25.455890 (+3320642us) ts_tablet_manager.cc:1151] Deleting tablet 60ae3db968f6412aa0c3952fafe65f88
1214 20:52:25.455899 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.455954 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3320514}
W20251214 20:52:25.456246 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8d9f74fb2c6244a6887e3c922262bbab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9f74fb2c6244a6887e3c922262bbab already in progress: creating tablet
I20251214 20:52:25.456422 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 216) took 3323 ms. Trace:
W20251214 20:52:25.456480 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 60ae3db968f6412aa0c3952fafe65f88 because the tablet was not found. No further retry: Not found: Tablet not found: 60ae3db968f6412aa0c3952fafe65f88
I20251214 20:52:25.456488 1856 rpcz_store.cc:276] 1214 20:52:22.132862 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.135393 (+ 2531us) service_pool.cc:225] Handling call
1214 20:52:25.456343 (+3320950us) ts_tablet_manager.cc:1151] Deleting tablet 8faa7fd7315f485c82aee38bcf4cd8fc
1214 20:52:25.456353 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.456408 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3320851}
W20251214 20:52:25.456825 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8faa7fd7315f485c82aee38bcf4cd8fc because the tablet was not found. No further retry: Not found: Tablet not found: 8faa7fd7315f485c82aee38bcf4cd8fc
I20251214 20:52:25.457077 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 203) took 3128 ms. Trace:
I20251214 20:52:25.457127 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 219) took 3324 ms. Trace:
I20251214 20:52:25.457162 1982 rpcz_store.cc:276] 1214 20:52:22.328708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.328922 (+ 214us) service_pool.cc:225] Handling call
1214 20:52:25.456677 (+3127755us) ts_tablet_manager.cc:1151] Deleting tablet fe24e4f4e74040249068505058c15622
1214 20:52:25.456973 (+ 296us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.457060 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3127588}
I20251214 20:52:25.457187 1856 rpcz_store.cc:276] 1214 20:52:22.133096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.135512 (+ 2416us) service_pool.cc:225] Handling call
1214 20:52:25.457018 (+3321506us) ts_tablet_manager.cc:1151] Deleting tablet 292a28b12df246a8bc855d81c530746e
1214 20:52:25.457027 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.457113 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3321394}
I20251214 20:52:25.457567 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 223) took 3324 ms. Trace:
W20251214 20:52:25.457556 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet fe24e4f4e74040249068505058c15622 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24e4f4e74040249068505058c15622 already in progress: creating tablet
I20251214 20:52:25.457629 1856 rpcz_store.cc:276] 1214 20:52:22.133386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.134221 (+ 835us) service_pool.cc:225] Handling call
1214 20:52:25.457488 (+3323267us) ts_tablet_manager.cc:1151] Deleting tablet 24ff58f321124d12b5330dca185d426e
1214 20:52:25.457498 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.457552 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3321308}
W20251214 20:52:25.457805 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 292a28b12df246a8bc855d81c530746e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292a28b12df246a8bc855d81c530746e already in progress: creating tablet
I20251214 20:52:25.457974 1856 tablet_replica.cc:333] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:25.458063 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 24ff58f321124d12b5330dca185d426e because the tablet was not found. No further retry: Not found: Tablet not found: 24ff58f321124d12b5330dca185d426e
I20251214 20:52:25.458178 1856 raft_consensus.cc:2243] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:25.458331 1856 raft_consensus.cc:2272] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:25.458356 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 201) took 3129 ms. Trace:
I20251214 20:52:25.458416 1982 rpcz_store.cc:276] 1214 20:52:22.328479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.329007 (+ 528us) service_pool.cc:225] Handling call
1214 20:52:25.458263 (+3129256us) ts_tablet_manager.cc:1151] Deleting tablet ba50e32f7e724868bb891045a939e362
1214 20:52:25.458273 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.458341 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3129003}
I20251214 20:52:25.458891 1856 ts_tablet_manager.cc:1916] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:25.458810 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ba50e32f7e724868bb891045a939e362 because the tablet was not found. No further retry: Not found: Tablet not found: ba50e32f7e724868bb891045a939e362
I20251214 20:52:25.461635 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 204) took 3132 ms. Trace:
I20251214 20:52:25.461721 1982 rpcz_store.cc:276] 1214 20:52:22.328807 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.329118 (+ 311us) service_pool.cc:225] Handling call
1214 20:52:25.461517 (+3132399us) ts_tablet_manager.cc:1151] Deleting tablet 65e79db74fa046698507c83416ee1669
1214 20:52:25.461527 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.461615 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3132203}
W20251214 20:52:25.462201 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 65e79db74fa046698507c83416ee1669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e79db74fa046698507c83416ee1669 already in progress: creating tablet
I20251214 20:52:25.462992 1202 tablet_bootstrap.cc:492] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:25.463251 1202 ts_tablet_manager.cc:1403] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20251214 20:52:25.464108 1202 raft_consensus.cc:359] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.464299 1202 raft_consensus.cc:385] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.464349 1202 raft_consensus.cc:740] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.464607 1202 consensus_queue.cc:260] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.469779 1202 ts_tablet_manager.cc:1434] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251214 20:52:25.476940 1202 tablet_bootstrap.cc:492] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:25.479081 1202 tablet_bootstrap.cc:654] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.481272 1206 tablet_bootstrap.cc:654] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.506067 1202 tablet_bootstrap.cc:492] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:25.506275 1202 ts_tablet_manager.cc:1403] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20251214 20:52:25.507277 1202 raft_consensus.cc:359] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.507484 1202 raft_consensus.cc:385] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.507532 1202 raft_consensus.cc:740] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.507848 1202 consensus_queue.cc:260] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.514918 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_fd97b3b527e240fbb6d62d3c33059904"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:25.515771 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd97b3b527e240fbb6d62d3c33059904 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:25.516824 1202 ts_tablet_manager.cc:1434] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20251214 20:52:25.519085 2368 raft_consensus.cc:493] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.519304 2368 raft_consensus.cc:515] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.520133 2368 leader_election.cc:290] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:25.520895 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec85857f047e44d78835305ddcb70768" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:25.521816 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec85857f047e44d78835305ddcb70768" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:25.522197 1202 tablet_bootstrap.cc:492] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:25.522312 991 leader_election.cc:343] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:25.522753 989 leader_election.cc:343] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.522853 989 leader_election.cc:304] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:25.523375 2368 raft_consensus.cc:2749] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:25.530673 1856 ts_tablet_manager.cc:1929] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:25.530797 1856 log.cc:1199] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/b315744c8cd043c5a8bdceec73a7e28b
I20251214 20:52:25.531104 1856 ts_tablet_manager.cc:1950] T b315744c8cd043c5a8bdceec73a7e28b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:25.531315 448 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet d1389716f5ce44f588c21a34c7a7d4c8
I20251214 20:52:25.531834 448 catalog_manager.cc:6724] ReplaceTablet: tablet d1389716f5ce44f588c21a34c7a7d4c8 deleted and replaced by tablet 894e3f27b7904db994711aa80a9b1600
I20251214 20:52:25.533123 2356 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:37530 (request call id 342) took 1089 ms. Trace:
I20251214 20:52:25.535451 2155 raft_consensus.cc:493] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.848805 2436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:34819 every 8 connection(s)
I20251214 20:52:25.849197 1769 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:34819
I20251214 20:52:25.873174 1769 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-1/data/info.pb
I20251214 20:52:25.535655 2155 raft_consensus.cc:515] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.538405 744 tablet_service.cc:1552] Processing DeleteTablet for tablet d1389716f5ce44f588c21a34c7a7d4c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 894e3f27b7904db994711aa80a9b1600) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:25.540582 768 raft_consensus.cc:3060] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:25.557897 1982 tablet_replica.cc:333] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:25.874755 2155 leader_election.cc:290] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:25.533243 2356 rpcz_store.cc:276] 1214 20:52:24.443273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.484914 (+ 41641us) service_pool.cc:225] Handling call
1214 20:52:25.533089 (+1048175us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:24.867408 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1214 20:52:24.867525 (+ 117us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1214 20:52:24.867534 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
1214 20:52:24.867546 (+ 12us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1214 20:52:24.867607 (+ 61us) alter_schema_op.cc:128] START. Timestamp: 1765745544867532
1214 20:52:24.867610 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:24.867817 (+ 207us) log.cc:844] Serialized 173 byte log entry
1214 20:52:25.095333 (+227516us) op_driver.cc:464] REPLICATION: finished
1214 20:52:25.251344 (+156011us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1214 20:52:25.324184 (+ 72840us) log.cc:844] Serialized 5 byte log entry
1214 20:52:25.532514 (+208330us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.532966 (+ 452us) log.cc:844] Serialized 14 byte log entry
1214 20:52:25.533013 (+ 47us) alter_schema_op.cc:73] Released schema lock
1214 20:52:25.533017 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":155948,"prepare.queue_time_us":43185,"prepare.run_cpu_time_us":11899,"prepare.run_wall_time_us":225587,"raft.queue_time_us":10693,"raft.run_cpu_time_us":775,"raft.run_wall_time_us":775,"replication_time_us":227692,"thread_start_us":400602,"threads_started":3,"wal-append.queue_time_us":219471}]]}
I20251214 20:52:25.539000 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet d1389716f5ce44f588c21a34c7a7d4c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 894e3f27b7904db994711aa80a9b1600) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:25.564083 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_624b9386a78b44e994cde78539ccedc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:25.875742 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu as pid 1769
W20251214 20:52:25.875837 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_624b9386a78b44e994cde78539ccedc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:25.541191 886 tablet_service.cc:1552] Processing DeleteTablet for tablet d1389716f5ce44f588c21a34c7a7d4c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 894e3f27b7904db994711aa80a9b1600) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:25.549638 1202 tablet_bootstrap.cc:654] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.565584 1203 tablet_bootstrap.cc:492] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:25.876536 1203 ts_tablet_manager.cc:1403] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.425s user 0.047s sys 0.049s
I20251214 20:52:25.582479 1650 raft_consensus.cc:493] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.877259 1650 raft_consensus.cc:515] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.877445 1203 raft_consensus.cc:359] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:25.877624 1203 raft_consensus.cc:385] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.877668 1203 raft_consensus.cc:740] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.877902 1650 leader_election.cc:290] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:25.877866 1203 consensus_queue.cc:260] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:25.624439 880 tablet_service.cc:1505] Processing CreateTablet for tablet e0d0e8f35bf44c9a8797b7507dcce3b9 (DEFAULT_TABLE table=default.Table_a4359e4ce4cb4a5caafc148f1dadef64 [id=75cebf8d0aba4aa48c1be51b0abd6b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.878297 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d0e8f35bf44c9a8797b7507dcce3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.878412 1203 ts_tablet_manager.cc:1434] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:25.878712 1203 tablet_bootstrap.cc:492] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:25.880626 1203 tablet_bootstrap.cc:654] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.581867 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet a142157acbcd44adb6099b4428103ee5
I20251214 20:52:25.881675 455 catalog_manager.cc:6724] ReplaceTablet: tablet a142157acbcd44adb6099b4428103ee5 deleted and replaced by tablet b2bea617954c475bac7e9b8719faa028
I20251214 20:52:25.881860 455 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 292) took 1098 ms. Trace:
I20251214 20:52:25.571833 1206 tablet_bootstrap.cc:492] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:25.882249 1206 ts_tablet_manager.cc:1403] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.501s user 0.004s sys 0.000s
I20251214 20:52:25.881953 455 rpcz_store.cc:276] 1214 20:52:24.783018 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.369039 (+586021us) service_pool.cc:225] Handling call
1214 20:52:25.389096 (+ 20057us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b18cf0b 00007ff411c97107 00007ff411c96e35 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:25.881789 (+492693us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:25.564877 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:25.565024 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:25.565030 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:25.565032 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:25.565185 (+ 153us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:25.565200 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:25.565205 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:25.565308 (+ 103us) tablet.cc:699] Row locks acquired
1214 20:52:25.565310 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:25.565351 (+ 41us) write_op.cc:270] Start()
1214 20:52:25.565390 (+ 39us) write_op.cc:276] Timestamp: P: 1765745545565255 usec, L: 0
1214 20:52:25.565394 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:25.565618 (+ 224us) log.cc:844] Serialized 656 byte log entry
1214 20:52:25.580521 (+ 14903us) op_driver.cc:464] REPLICATION: finished
1214 20:52:25.580610 (+ 89us) write_op.cc:301] APPLY: starting
1214 20:52:25.580628 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:25.580673 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:25.580704 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:25.580810 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:25.580890 (+ 80us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:25.580898 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:25.580974 (+ 76us) log.cc:844] Serialized 29 byte log entry
1214 20:52:25.581033 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:25.581051 (+ 18us) write_op.cc:454] Released schema lock
1214 20:52:25.581058 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:25.581130 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:25.581134 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":45292416,"child_traces":[["op",{"apply.queue_time_us":31,"apply.run_cpu_time_us":513,"apply.run_wall_time_us":514,"num_ops":2,"prepare.queue_time_us":174796,"prepare.run_cpu_time_us":962,"prepare.run_wall_time_us":8944,"raft.queue_time_us":109,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":289,"replication_time_us":15091}]]}
I20251214 20:52:25.882822 455 master_service.cc:954] ReplaceTablet: received request to replace tablet 57a6232564194cbba12182f3a29c2ebb from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:25.883118 1206 raft_consensus.cc:359] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.883324 1206 raft_consensus.cc:385] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.883378 1206 raft_consensus.cc:740] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.624725 877 tablet_service.cc:1505] Processing CreateTablet for tablet 899f21bafb3c4db9b23907cdc9f121d8 (DEFAULT_TABLE table=default.Table_e95c404ef11b4cf79b5a81ec1eddb57a [id=fb52e77ee3ec48c39afd588c8de11764]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.883579 1206 consensus_queue.cc:260] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.883981 1206 ts_tablet_manager.cc:1434] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:25.883992 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899f21bafb3c4db9b23907cdc9f121d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.884346 1206 tablet_bootstrap.cc:492] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:25.582836 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 224) took 3435 ms. Trace:
I20251214 20:52:25.885104 1856 rpcz_store.cc:276] 1214 20:52:22.147265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.147335 (+ 70us) service_pool.cc:225] Handling call
1214 20:52:25.457927 (+3310592us) ts_tablet_manager.cc:1151] Deleting tablet b315744c8cd043c5a8bdceec73a7e28b
1214 20:52:25.457936 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.527702 (+ 69766us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.530571 (+ 2869us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.582810 (+ 52239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3310408}
I20251214 20:52:25.885620 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 225) took 3736 ms. Trace:
I20251214 20:52:25.885694 1856 rpcz_store.cc:276] 1214 20:52:22.149039 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.149088 (+ 49us) service_pool.cc:225] Handling call
1214 20:52:25.885468 (+3736380us) ts_tablet_manager.cc:1151] Deleting tablet c16895ca08d94b56b5c4bc9352e330ef
1214 20:52:25.885480 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.885596 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3736182}
I20251214 20:52:25.885826 1206 tablet_bootstrap.cc:654] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.885982 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 226) took 3736 ms. Trace:
I20251214 20:52:25.886054 1856 rpcz_store.cc:276] 1214 20:52:22.149420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.149460 (+ 40us) service_pool.cc:225] Handling call
1214 20:52:25.885891 (+3736431us) ts_tablet_manager.cc:1151] Deleting tablet a34eb8fa3e34427495fa5985b7ba81bf
1214 20:52:25.885899 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.885964 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3736287}
I20251214 20:52:25.886317 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 227) took 3736 ms. Trace:
I20251214 20:52:25.886373 1856 rpcz_store.cc:276] 1214 20:52:22.149747 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.149796 (+ 49us) service_pool.cc:225] Handling call
1214 20:52:25.886234 (+3736438us) ts_tablet_manager.cc:1151] Deleting tablet 25974e5db555447baf5799775d15cd14
1214 20:52:25.886243 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.886291 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3736312}
I20251214 20:52:25.886624 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 228) took 3731 ms. Trace:
I20251214 20:52:25.886680 1856 rpcz_store.cc:276] 1214 20:52:22.155568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.155620 (+ 52us) service_pool.cc:225] Handling call
1214 20:52:25.886542 (+3730922us) ts_tablet_manager.cc:1151] Deleting tablet 3c3962c33a294fa98f8bd485268aee71
1214 20:52:25.886549 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.886611 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3730758}
I20251214 20:52:25.886886 1856 tablet_replica.cc:333] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:25.887079 1856 raft_consensus.cc:2243] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:25.628319 887 tablet_service.cc:1505] Processing CreateTablet for tablet 365dc8e0324c415fb35d857849911595 (DEFAULT_TABLE table=default.Table_c96142a94af44f24bb7c6e51bf5e1f3e [id=296657ce558f4c599fc2013a71b7bda3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.887494 1856 raft_consensus.cc:2272] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:25.887552 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365dc8e0324c415fb35d857849911595. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.888123 1856 ts_tablet_manager.cc:1916] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:25.631069 742 tablet_service.cc:1505] Processing CreateTablet for tablet e0d0e8f35bf44c9a8797b7507dcce3b9 (DEFAULT_TABLE table=default.Table_a4359e4ce4cb4a5caafc148f1dadef64 [id=75cebf8d0aba4aa48c1be51b0abd6b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.889163 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d0e8f35bf44c9a8797b7507dcce3b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:25.582247 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4643effa71d4d18850913a59ae1c75d consensus state has no leader
I20251214 20:52:25.622828 1021 tablet_service.cc:1505] Processing CreateTablet for tablet e0d0e8f35bf44c9a8797b7507dcce3b9 (DEFAULT_TABLE table=default.Table_a4359e4ce4cb4a5caafc148f1dadef64 [id=75cebf8d0aba4aa48c1be51b0abd6b38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.630187 889 tablet_service.cc:1505] Processing CreateTablet for tablet b67190c21429415081c867332b080f73 (DEFAULT_TABLE table=default.Table_183ef00a0ed14492bb36f72ed1c73aa2 [id=a61d5b8118774bb2b806d755ab98da86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.891664 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_2541d743bd8945a2b0d984cdf911d6ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:25.892108 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b67190c21429415081c867332b080f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.892041 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16816fa6b8c34207bb929147910b93cb" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:25.892314 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2541d743bd8945a2b0d984cdf911d6ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:25.582659 454 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (table default.Table_89c0478b97144f35b1a0adeb64be2d9f [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]). Expected version 1 got 0
I20251214 20:52:25.895948 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet a142157acbcd44adb6099b4428103ee5 with delete_type TABLET_DATA_DELETED (replaced by tablet b2bea617954c475bac7e9b8719faa028) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:25.896060 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbbfcef827ee4d548939aaf4783c8109" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:25.636634 749 tablet_service.cc:1505] Processing CreateTablet for tablet 899f21bafb3c4db9b23907cdc9f121d8 (DEFAULT_TABLE table=default.Table_e95c404ef11b4cf79b5a81ec1eddb57a [id=fb52e77ee3ec48c39afd588c8de11764]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.594309 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_5b1cf9007e0b4c3bae65a57ec193d8fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:25.896832 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899f21bafb3c4db9b23907cdc9f121d8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:25.896934 851 leader_election.cc:343] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:25.897097 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b1cf9007e0b4c3bae65a57ec193d8fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:25.901499 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet b315744c8cd043c5a8bdceec73a7e28b (table default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]) successfully deleted
W20251214 20:52:25.901770 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c16895ca08d94b56b5c4bc9352e330ef because the tablet was not found. No further retry: Not found: Tablet not found: c16895ca08d94b56b5c4bc9352e330ef
W20251214 20:52:25.901932 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a34eb8fa3e34427495fa5985b7ba81bf because the tablet was not found. No further retry: Not found: Tablet not found: a34eb8fa3e34427495fa5985b7ba81bf
W20251214 20:52:25.902091 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 25974e5db555447baf5799775d15cd14 because the tablet was not found. No further retry: Not found: Tablet not found: 25974e5db555447baf5799775d15cd14
W20251214 20:52:25.902292 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3c3962c33a294fa98f8bd485268aee71 because the tablet was not found. No further retry: Not found: Tablet not found: 3c3962c33a294fa98f8bd485268aee71
I20251214 20:52:25.902415 1202 tablet_bootstrap.cc:492] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:25.902573 1202 ts_tablet_manager.cc:1403] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.382s user 0.003s sys 0.001s
I20251214 20:52:25.623175 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 899f21bafb3c4db9b23907cdc9f121d8 (DEFAULT_TABLE table=default.Table_e95c404ef11b4cf79b5a81ec1eddb57a [id=fb52e77ee3ec48c39afd588c8de11764]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.903195 886 tablet_service.cc:1552] Processing DeleteTablet for tablet a142157acbcd44adb6099b4428103ee5 with delete_type TABLET_DATA_DELETED (replaced by tablet b2bea617954c475bac7e9b8719faa028) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:25.635818 878 tablet_service.cc:1505] Processing CreateTablet for tablet bea3e9d7334345fa88ededf1be17a434 (DEFAULT_TABLE table=default.Table_27e3193bffa748be95f7c923662ad899 [id=e24a6c9e5c9049edba5e42e22d22a831]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.903434 1202 raft_consensus.cc:359] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.903611 1202 raft_consensus.cc:385] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:25.903656 1202 raft_consensus.cc:740] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:25.903766 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea3e9d7334345fa88ededf1be17a434. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.903848 1202 consensus_queue.cc:260] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.904367 1202 ts_tablet_manager.cc:1434] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:25.907964 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:25.908200 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]). Expected version 1 got 0
I20251214 20:52:25.908722 1982 raft_consensus.cc:2243] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:25.908963 1982 raft_consensus.cc:2272] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:25.637283 745 tablet_service.cc:1505] Processing CreateTablet for tablet b67190c21429415081c867332b080f73 (DEFAULT_TABLE table=default.Table_183ef00a0ed14492bb36f72ed1c73aa2 [id=a61d5b8118774bb2b806d755ab98da86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.909621 1982 ts_tablet_manager.cc:1916] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:25.909922 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b67190c21429415081c867332b080f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.637485 741 tablet_service.cc:1505] Processing CreateTablet for tablet bea3e9d7334345fa88ededf1be17a434 (DEFAULT_TABLE table=default.Table_27e3193bffa748be95f7c923662ad899 [id=e24a6c9e5c9049edba5e42e22d22a831]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.910707 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea3e9d7334345fa88ededf1be17a434. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.913038 2431 raft_consensus.cc:493] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.913204 2431 raft_consensus.cc:515] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.913928 2431 leader_election.cc:290] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:25.657312 452 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 1eff8a23d48c43e2bdbfb34f4bab6edf
I20251214 20:52:25.914613 452 catalog_manager.cc:6724] ReplaceTablet: tablet 1eff8a23d48c43e2bdbfb34f4bab6edf deleted and replaced by tablet ed9790fbebda4f848291930bf1dd8174
I20251214 20:52:25.914752 452 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50088 (request call id 291) took 1223 ms. Trace:
I20251214 20:52:25.637632 753 tablet_service.cc:1505] Processing CreateTablet for tablet 76e98dbef93c437c83db3b6dbdb06afa (DEFAULT_TABLE table=default.Table_58619bc0f4b54a208cabd270329ccf82 [id=3f4e5f296df7465b9c9ef0c86e7e2b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.637723 747 tablet_service.cc:1505] Processing CreateTablet for tablet 365dc8e0324c415fb35d857849911595 (DEFAULT_TABLE table=default.Table_c96142a94af44f24bb7c6e51bf5e1f3e [id=296657ce558f4c599fc2013a71b7bda3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.914847 452 rpcz_store.cc:276] 1214 20:52:24.691733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.349913 (+658180us) service_pool.cc:225] Handling call
1214 20:52:25.361516 (+ 11603us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b18cf0b 00007ff411c97107 00007ff411c96e35 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:25.914711 (+553195us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:25.644822 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:25.644915 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:25.644920 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:25.644923 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:25.645037 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:25.645051 (+ 14us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:25.645056 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:25.645154 (+ 98us) tablet.cc:699] Row locks acquired
1214 20:52:25.645156 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:25.645185 (+ 29us) write_op.cc:270] Start()
1214 20:52:25.645354 (+ 169us) write_op.cc:276] Timestamp: P: 1765745545645078 usec, L: 0
1214 20:52:25.645357 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:25.645606 (+ 249us) log.cc:844] Serialized 622 byte log entry
1214 20:52:25.655643 (+ 10037us) op_driver.cc:464] REPLICATION: finished
1214 20:52:25.656100 (+ 457us) write_op.cc:301] APPLY: starting
1214 20:52:25.656125 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:25.656163 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:25.656196 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:25.656298 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:25.656349 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:25.656355 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:25.656418 (+ 63us) log.cc:844] Serialized 29 byte log entry
1214 20:52:25.656470 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:25.656487 (+ 17us) write_op.cc:454] Released schema lock
1214 20:52:25.656493 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:25.656551 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:25.656554 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":25653504,"child_traces":[["op",{"apply.queue_time_us":415,"apply.run_cpu_time_us":445,"apply.run_wall_time_us":444,"num_ops":2,"prepare.queue_time_us":24647,"prepare.run_cpu_time_us":774,"prepare.run_wall_time_us":2443,"raft.queue_time_us":66,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":231,"replication_time_us":10253}]]}
I20251214 20:52:25.915423 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76e98dbef93c437c83db3b6dbdb06afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.915630 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365dc8e0324c415fb35d857849911595. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.915783 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_4bb4b496a62d4c89b7b4e783023dd5cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:25.916406 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bb4b496a62d4c89b7b4e783023dd5cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:25.623433 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 365dc8e0324c415fb35d857849911595 (DEFAULT_TABLE table=default.Table_c96142a94af44f24bb7c6e51bf5e1f3e [id=296657ce558f4c599fc2013a71b7bda3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.636036 884 tablet_service.cc:1505] Processing CreateTablet for tablet 30595c2e1f814a66b29722caf123edce (DEFAULT_TABLE table=default.Table_db687aab85f2411f8433eb14724da9ac [id=83ef50f71fb34ecda09e60f00f8ec8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.917945 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30595c2e1f814a66b29722caf123edce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.636202 885 tablet_service.cc:1505] Processing CreateTablet for tablet 76e98dbef93c437c83db3b6dbdb06afa (DEFAULT_TABLE table=default.Table_58619bc0f4b54a208cabd270329ccf82 [id=3f4e5f296df7465b9c9ef0c86e7e2b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.918478 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76e98dbef93c437c83db3b6dbdb06afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.637786 750 tablet_service.cc:1505] Processing CreateTablet for tablet 30595c2e1f814a66b29722caf123edce (DEFAULT_TABLE table=default.Table_db687aab85f2411f8433eb14724da9ac [id=83ef50f71fb34ecda09e60f00f8ec8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.919377 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30595c2e1f814a66b29722caf123edce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.922454 1982 ts_tablet_manager.cc:1929] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:25.922566 1982 log.cc:1199] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/35a529702f924a5c9bf53c3c668c6094
I20251214 20:52:25.922878 1982 ts_tablet_manager.cc:1950] T 35a529702f924a5c9bf53c3c668c6094 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:25.638111 746 tablet_service.cc:1505] Processing CreateTablet for tablet b8727f302d35471eb2eac1e77b8d966b (DEFAULT_TABLE table=default.Table_01b1f692968d4dbe8d92746b6017a8b3 [id=eefab419a1984e92997ff7429a6dd368]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.923518 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8727f302d35471eb2eac1e77b8d966b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.925200 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899f21bafb3c4db9b23907cdc9f121d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.924978 447 catalog_manager.cc:5654] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:25.925693 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]). Expected version 2 got 0
I20251214 20:52:25.925693 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738668adf2874d6894db6fa875f6990c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:25.925913 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:25.926167 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:25.926503 447 catalog_manager.cc:5654] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:25.638698 743 tablet_service.cc:1505] Processing CreateTablet for tablet cbefd8f111084785bafd7e74ac443fd0 (DEFAULT_TABLE table=default.Table_9ba7f3a5c10e401b9edd530f37db3e17 [id=d9dec831289f4420b60b64589ee35fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.657593 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 2f97e5bad46343098418e8e34334b892
I20251214 20:52:25.927155 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbefd8f111084785bafd7e74ac443fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.927598 1202 tablet_bootstrap.cc:492] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:25.928102 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0" } modify_external_catalogs: true
I20251214 20:52:25.928270 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0" } modify_external_catalogs: true
I20251214 20:52:25.629264 1023 tablet_service.cc:1505] Processing CreateTablet for tablet b67190c21429415081c867332b080f73 (DEFAULT_TABLE table=default.Table_183ef00a0ed14492bb36f72ed1c73aa2 [id=a61d5b8118774bb2b806d755ab98da86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.637769 879 tablet_service.cc:1505] Processing CreateTablet for tablet b8727f302d35471eb2eac1e77b8d966b (DEFAULT_TABLE table=default.Table_01b1f692968d4dbe8d92746b6017a8b3 [id=eefab419a1984e92997ff7429a6dd368]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.929168 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8727f302d35471eb2eac1e77b8d966b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.929234 1202 tablet_bootstrap.cc:654] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:25.640978 748 tablet_service.cc:1505] Processing CreateTablet for tablet 1c82068013e8471c9309bcf8e8604ece (DEFAULT_TABLE table=default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.664036 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet c9488c3505f5443998531cf749722be9
I20251214 20:52:25.931365 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c82068013e8471c9309bcf8e8604ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.932045 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_7e0acfc6d6e84dad84938bb81b1f375d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:25.932768 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e0acfc6d6e84dad84938bb81b1f375d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:25.932883 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 208) took 3603 ms. Trace:
I20251214 20:52:25.932972 1982 rpcz_store.cc:276] 1214 20:52:22.329228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.329340 (+ 112us) service_pool.cc:225] Handling call
1214 20:52:25.557818 (+3228478us) ts_tablet_manager.cc:1151] Deleting tablet 35a529702f924a5c9bf53c3c668c6094
1214 20:52:25.557833 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.913990 (+356157us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.922350 (+ 8360us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.932858 (+ 10508us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34346,"tablet-delete.queue_time_us":3228338}
I20251214 20:52:25.933391 1982 tablet_replica.cc:333] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:25.933590 1982 raft_consensus.cc:2243] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:25.933754 1982 raft_consensus.cc:2272] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:25.631507 1030 tablet_service.cc:1505] Processing CreateTablet for tablet bea3e9d7334345fa88ededf1be17a434 (DEFAULT_TABLE table=default.Table_27e3193bffa748be95f7c923662ad899 [id=e24a6c9e5c9049edba5e42e22d22a831]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.633194 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 76e98dbef93c437c83db3b6dbdb06afa (DEFAULT_TABLE table=default.Table_58619bc0f4b54a208cabd270329ccf82 [id=3f4e5f296df7465b9c9ef0c86e7e2b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.634999 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 30595c2e1f814a66b29722caf123edce (DEFAULT_TABLE table=default.Table_db687aab85f2411f8433eb14724da9ac [id=83ef50f71fb34ecda09e60f00f8ec8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.636920 1029 tablet_service.cc:1505] Processing CreateTablet for tablet b8727f302d35471eb2eac1e77b8d966b (DEFAULT_TABLE table=default.Table_01b1f692968d4dbe8d92746b6017a8b3 [id=eefab419a1984e92997ff7429a6dd368]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.639695 1028 tablet_service.cc:1505] Processing CreateTablet for tablet cbefd8f111084785bafd7e74ac443fd0 (DEFAULT_TABLE table=default.Table_9ba7f3a5c10e401b9edd530f37db3e17 [id=d9dec831289f4420b60b64589ee35fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.640468 882 tablet_service.cc:1505] Processing CreateTablet for tablet cbefd8f111084785bafd7e74ac443fd0 (DEFAULT_TABLE table=default.Table_9ba7f3a5c10e401b9edd530f37db3e17 [id=d9dec831289f4420b60b64589ee35fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.641088 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 1c82068013e8471c9309bcf8e8604ece (DEFAULT_TABLE table=default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.642838 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 6c8af3d2720a499eb51bcfb213bcdbd1 (DEFAULT_TABLE table=default.Table_ae25c010c04548198844cd6f5f3a5693 [id=7159a40b641e406d89e8a11b2b1eba82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.935472 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbefd8f111084785bafd7e74ac443fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.935766 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d0e8f35bf44c9a8797b7507dcce3b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.642508 756 tablet_service.cc:1505] Processing CreateTablet for tablet 6c8af3d2720a499eb51bcfb213bcdbd1 (DEFAULT_TABLE table=default.Table_ae25c010c04548198844cd6f5f3a5693 [id=7159a40b641e406d89e8a11b2b1eba82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.936106 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8af3d2720a499eb51bcfb213bcdbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.936417 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365dc8e0324c415fb35d857849911595. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.641654 890 tablet_service.cc:1505] Processing CreateTablet for tablet 1c82068013e8471c9309bcf8e8604ece (DEFAULT_TABLE table=default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.939446 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c82068013e8471c9309bcf8e8604ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.669615 2368 raft_consensus.cc:493] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.713398 2167 raft_consensus.cc:493] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.816228 2364 raft_consensus.cc:493] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.939932 2368 raft_consensus.cc:515] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.940040 2364 raft_consensus.cc:515] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:25.749188 713 consensus_peers.cc:597] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 762068d560ff47918dd9fd8791083433. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:25.939957 2167 raft_consensus.cc:515] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:25.940884 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 191747us
I20251214 20:52:25.941197 2167 leader_election.cc:290] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:25.940707 2364 leader_election.cc:290] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:25.940711 2368 leader_election.cc:290] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:25.643766 883 tablet_service.cc:1505] Processing CreateTablet for tablet 6c8af3d2720a499eb51bcfb213bcdbd1 (DEFAULT_TABLE table=default.Table_ae25c010c04548198844cd6f5f3a5693 [id=7159a40b641e406d89e8a11b2b1eba82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:25.942032 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8af3d2720a499eb51bcfb213bcdbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.648129 2287 raft_consensus.cc:493] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:25.942451 2287 raft_consensus.cc:515] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.943048 2287 leader_election.cc:290] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:25.664736 1977 ts_tablet_manager.cc:1929] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:25.943634 1977 log.cc:1199] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/6381cbf05a8c4f84b71f229497fd28ef
W20251214 20:52:25.786795 848 consensus_peers.cc:597] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:25.943807 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 157033us
I20251214 20:52:25.944991 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf with delete_type TABLET_DATA_DELETED (replaced by tablet ed9790fbebda4f848291930bf1dd8174) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:25.945225 881 tablet_service.cc:1552] Processing DeleteTablet for tablet c9488c3505f5443998531cf749722be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:25.945204 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f97e5bad46343098418e8e34334b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:25.946182 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbbfcef827ee4d548939aaf4783c8109" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:25.946569 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98105bceda9b44bf893f82a4234a3c81" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:25.947777 848 leader_election.cc:343] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.947893 848 leader_election.cc:304] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:25.948336 848 leader_election.cc:343] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.948757 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c82068013e8471c9309bcf8e8604ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.948978 1650 raft_consensus.cc:2749] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:25.949698 991 leader_election.cc:343] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.950019 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_13184fd4bbc74e5f8bea847a1c8674eb" } new_table_name: "default.Table_0945dd902049447298aef35db32f33ba" modify_external_catalogs: true
I20251214 20:52:25.950106 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98105bceda9b44bf893f82a4234a3c81" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:25.950263 915 raft_consensus.cc:2468] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:25.950547 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f198ef4dfa4f5791144b9a24fe693f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:25.950801 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d164747cb294bd4b126b4420231867d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:25.950922 904 raft_consensus.cc:2468] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:25.951367 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f97e5bad46343098418e8e34334b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:25.951460 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "16816fa6b8c34207bb929147910b93cb" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:25.951385 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet c9488c3505f5443998531cf749722be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:25.951666 1050 raft_consensus.cc:2393] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate a2e8fd1cb731446d88f8e510c7a52999 in this term.
I20251214 20:52:25.951723 991 leader_election.cc:304] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:25.951511 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbefd8f111084785bafd7e74ac443fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.951948 2368 raft_consensus.cc:2804] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:25.952032 2368 raft_consensus.cc:493] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:25.952106 2368 raft_consensus.cc:3060] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:25.952191 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8727f302d35471eb2eac1e77b8d966b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.952534 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30595c2e1f814a66b29722caf123edce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.952870 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8af3d2720a499eb51bcfb213bcdbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.953202 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76e98dbef93c437c83db3b6dbdb06afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.953516 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b67190c21429415081c867332b080f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.953879 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea3e9d7334345fa88ededf1be17a434. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:25.954744 2368 raft_consensus.cc:515] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.955397 2368 leader_election.cc:290] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:25.951960 991 leader_election.cc:343] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:25.955753 991 leader_election.cc:304] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:25.956274 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98105bceda9b44bf893f82a4234a3c81" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:25.956516 1050 raft_consensus.cc:2393] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2e8fd1cb731446d88f8e510c7a52999 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:25.956476 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98105bceda9b44bf893f82a4234a3c81" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:25.956662 904 raft_consensus.cc:3060] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:25.956964 2368 raft_consensus.cc:2804] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:25.957033 2368 raft_consensus.cc:493] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:25.957165 2368 raft_consensus.cc:3060] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:25.958344 851 leader_election.cc:304] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:25.960952 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf with delete_type TABLET_DATA_DELETED (replaced by tablet ed9790fbebda4f848291930bf1dd8174) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:25.961333 1977 ts_tablet_manager.cc:1950] T 6381cbf05a8c4f84b71f229497fd28ef P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:25.961432 2368 raft_consensus.cc:515] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.962100 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 197) took 2968 ms. Trace:
I20251214 20:52:25.962178 1977 rpcz_store.cc:276] 1214 20:52:22.994073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.994115 (+ 42us) service_pool.cc:225] Handling call
1214 20:52:25.375475 (+2381360us) ts_tablet_manager.cc:1151] Deleting tablet 6381cbf05a8c4f84b71f229497fd28ef
1214 20:52:25.375494 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.631567 (+256073us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.664360 (+ 32793us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.962077 (+297717us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2381214}
I20251214 20:52:25.962611 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 198) took 2968 ms. Trace:
I20251214 20:52:25.962672 1977 rpcz_store.cc:276] 1214 20:52:22.994420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.994463 (+ 43us) service_pool.cc:225] Handling call
1214 20:52:25.962452 (+2967989us) ts_tablet_manager.cc:1151] Deleting tablet 98b7cac3930a4a9eb16ac413c8f1e092
1214 20:52:25.962475 (+ 23us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.962594 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2967847}
I20251214 20:52:25.962797 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d164747cb294bd4b126b4420231867d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:25.962903 1977 tablet_replica.cc:333] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:25.962985 903 raft_consensus.cc:3060] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:25.963099 1977 raft_consensus.cc:2243] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:25.963274 1977 raft_consensus.cc:2272] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:25.963842 1977 ts_tablet_manager.cc:1916] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:25.965737 903 raft_consensus.cc:2468] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:25.966225 991 leader_election.cc:304] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:25.966531 2364 raft_consensus.cc:2804] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:25.966634 2364 raft_consensus.cc:697] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:25.966876 2364 consensus_queue.cc:237] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.969609 904 raft_consensus.cc:2468] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:25.970398 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 6381cbf05a8c4f84b71f229497fd28ef (table default.Table_035a23623ce242298ab89c51024d7a09 [id=cd7616f6ff3244ec8628da5102e51570]) successfully deleted
W20251214 20:52:25.970713 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 98b7cac3930a4a9eb16ac413c8f1e092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98b7cac3930a4a9eb16ac413c8f1e092 already in progress: creating tablet
I20251214 20:52:25.970885 2287 raft_consensus.cc:2749] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:25.971393 991 leader_election.cc:304] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:25.971686 2167 raft_consensus.cc:2804] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:25.971797 2167 raft_consensus.cc:697] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:25.972002 2167 consensus_queue.cc:237] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:25.974531 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 57a6232564194cbba12182f3a29c2ebb
I20251214 20:52:25.975436 455 catalog_manager.cc:6724] ReplaceTablet: tablet 57a6232564194cbba12182f3a29c2ebb deleted and replaced by tablet aaf060ab2f71429a85dda19669e69677
I20251214 20:52:25.976634 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ed823cd2c6d94ed2b4707f427c8a2867" } new_table_name: "default.Table_adc1626f17674e3cab49c72818eec7d6" modify_external_catalogs: true
I20251214 20:52:25.976927 1977 ts_tablet_manager.cc:1929] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:25.977034 1977 log.cc:1199] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/4b86f4adaf4540f48a4f42bc45e259e5
I20251214 20:52:25.977352 1977 ts_tablet_manager.cc:1950] T 4b86f4adaf4540f48a4f42bc45e259e5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:25.978067 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 199) took 2983 ms. Trace:
I20251214 20:52:25.978154 1977 rpcz_store.cc:276] 1214 20:52:22.994709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.994749 (+ 40us) service_pool.cc:225] Handling call
1214 20:52:25.962862 (+2968113us) ts_tablet_manager.cc:1151] Deleting tablet 4b86f4adaf4540f48a4f42bc45e259e5
1214 20:52:25.962871 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.966745 (+ 3874us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.976837 (+ 10092us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:25.978048 (+ 1211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2967981}
I20251214 20:52:25.978433 1977 tablet_replica.cc:333] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:25.978602 1977 raft_consensus.cc:2243] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:25.978739 1977 raft_consensus.cc:2272] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:25.978749 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 4b86f4adaf4540f48a4f42bc45e259e5 (table default.Table_69c0719fe94a4c7594516a739f6a40ea [id=a5424dd4eee64d0686d5e4a8a5c35213]) successfully deleted
I20251214 20:52:25.984083 2368 leader_election.cc:290] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:25.986172 1977 ts_tablet_manager.cc:1916] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:25.991657 1982 ts_tablet_manager.cc:1916] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.012305 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 57a6232564194cbba12182f3a29c2ebb with delete_type TABLET_DATA_DELETED (replaced by tablet aaf060ab2f71429a85dda19669e69677) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.013053 1856 ts_tablet_manager.cc:1929] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.013293 1856 log.cc:1199] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ee7d709316b24b8eb223a838710c2979
I20251214 20:52:26.013670 1856 ts_tablet_manager.cc:1950] T ee7d709316b24b8eb223a838710c2979 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.015030 2155 consensus_queue.cc:1048] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:26.015568 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 229) took 3859 ms. Trace:
I20251214 20:52:26.015727 1856 rpcz_store.cc:276] 1214 20:52:22.156056 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.156092 (+ 36us) service_pool.cc:225] Handling call
1214 20:52:25.886837 (+3730745us) ts_tablet_manager.cc:1151] Deleting tablet ee7d709316b24b8eb223a838710c2979
1214 20:52:25.886844 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.925872 (+ 39028us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.012821 (+ 86949us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.015548 (+ 2727us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3730589}
I20251214 20:52:26.019411 2368 consensus_queue.cc:1048] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:26.028568 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ee7d709316b24b8eb223a838710c2979 (table default.Table_aa507956e1b54af68730c20f7c10e842 [id=f249932331aa4016806e482e8c7c4179]) successfully deleted
I20251214 20:52:26.041476 2327 consensus_queue.cc:1048] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:26.046223 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 57a6232564194cbba12182f3a29c2ebb with delete_type TABLET_DATA_DELETED (replaced by tablet aaf060ab2f71429a85dda19669e69677) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.055490 1977 ts_tablet_manager.cc:1929] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.055807 1977 log.cc:1199] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ba50e32f7e724868bb891045a939e362
I20251214 20:52:26.056223 1977 ts_tablet_manager.cc:1950] T ba50e32f7e724868bb891045a939e362 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.057046 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 200) took 3062 ms. Trace:
I20251214 20:52:26.061707 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 35a529702f924a5c9bf53c3c668c6094 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]) successfully deleted
I20251214 20:52:26.061790 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf with delete_type TABLET_DATA_DELETED (replaced by tablet ed9790fbebda4f848291930bf1dd8174) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.062126 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f97e5bad46343098418e8e34334b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.062354 749 tablet_service.cc:1552] Processing DeleteTablet for tablet a142157acbcd44adb6099b4428103ee5 with delete_type TABLET_DATA_DELETED (replaced by tablet b2bea617954c475bac7e9b8719faa028) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.072402 749 tablet_service.cc:1552] Processing DeleteTablet for tablet c9488c3505f5443998531cf749722be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.073398 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 57a6232564194cbba12182f3a29c2ebb with delete_type TABLET_DATA_DELETED (replaced by tablet aaf060ab2f71429a85dda19669e69677) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.076265 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738668adf2874d6894db6fa875f6990c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:26.077149 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98105bceda9b44bf893f82a4234a3c81" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:26.077958 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f198ef4dfa4f5791144b9a24fe693f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:26.078675 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d164747cb294bd4b126b4420231867d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:26.079411 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98105bceda9b44bf893f82a4234a3c81" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:26.084120 1977 rpcz_store.cc:276] 1214 20:52:22.995051 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.995094 (+ 43us) service_pool.cc:225] Handling call
1214 20:52:25.978391 (+2983297us) ts_tablet_manager.cc:1151] Deleting tablet ba50e32f7e724868bb891045a939e362
1214 20:52:25.978400 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:25.990168 (+ 11768us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.055344 (+ 65176us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.057024 (+ 1680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2983136}
I20251214 20:52:26.084609 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 201) took 3089 ms. Trace:
I20251214 20:52:26.084694 1977 rpcz_store.cc:276] 1214 20:52:22.995444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.995489 (+ 45us) service_pool.cc:225] Handling call
1214 20:52:26.084471 (+3088982us) ts_tablet_manager.cc:1151] Deleting tablet c9b7149a436a4cab80b39adccf8567f9
1214 20:52:26.084483 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.084591 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3088844}
I20251214 20:52:26.084968 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 177) took 3110 ms. Trace:
I20251214 20:52:26.085041 1977 rpcz_store.cc:276] 1214 20:52:22.974367 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.999343 (+ 24976us) service_pool.cc:225] Handling call
1214 20:52:26.084885 (+3085542us) ts_tablet_manager.cc:1151] Deleting tablet 2842f1dc0c374d24946f90dac1cf8aa5
1214 20:52:26.084894 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.084952 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085308}
I20251214 20:52:26.085364 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 202) took 3082 ms. Trace:
I20251214 20:52:26.085435 1977 rpcz_store.cc:276] 1214 20:52:23.003044 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.003171 (+ 127us) service_pool.cc:225] Handling call
1214 20:52:26.085267 (+3082096us) ts_tablet_manager.cc:1151] Deleting tablet 3c3962c33a294fa98f8bd485268aee71
1214 20:52:26.085276 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.085353 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3081478}
I20251214 20:52:26.085692 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 203) took 3081 ms. Trace:
I20251214 20:52:26.085757 1977 rpcz_store.cc:276] 1214 20:52:23.004469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.004513 (+ 44us) service_pool.cc:225] Handling call
1214 20:52:26.085601 (+3081088us) ts_tablet_manager.cc:1151] Deleting tablet bf8643c7e4094f529cd55656c7b3edb6
1214 20:52:26.085608 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.085680 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3080945}
I20251214 20:52:26.085999 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 206) took 3075 ms. Trace:
I20251214 20:52:26.086066 1977 rpcz_store.cc:276] 1214 20:52:23.010868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.010910 (+ 42us) service_pool.cc:225] Handling call
1214 20:52:26.085916 (+3075006us) ts_tablet_manager.cc:1151] Deleting tablet 02e256d4537249a394b80dc36beefef4
1214 20:52:26.085923 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.085987 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3074838}
I20251214 20:52:26.086318 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 207) took 3073 ms. Trace:
I20251214 20:52:26.063946 1202 tablet_bootstrap.cc:492] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:26.087071 1202 ts_tablet_manager.cc:1403] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.160s user 0.004s sys 0.015s
I20251214 20:52:26.087909 1202 raft_consensus.cc:359] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.087929 454 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd] alter complete (version 1)
I20251214 20:52:26.088075 1202 raft_consensus.cc:385] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.088128 1202 raft_consensus.cc:740] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.088295 1202 consensus_queue.cc:260] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.089056 1202 ts_tablet_manager.cc:1434] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:26.089244 446 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 298) took 1136 ms. Trace:
I20251214 20:52:26.089329 1202 tablet_bootstrap.cc:492] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:26.089393 446 rpcz_store.cc:276] 1214 20:52:24.952986 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.591414 (+638428us) service_pool.cc:225] Handling call
1214 20:52:25.594256 (+ 2842us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b1801c3 00007ff411c8a7c6 00007ff411c8a4f5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:25.897196 (+302940us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:25.897668 (+ 472us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:26.088950 (+191282us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:26.089064 (+ 114us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:26.089181 (+ 117us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:25.916857 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:25.916945 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:25.916950 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:25.916952 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:25.917063 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:25.917071 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:25.917075 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:25.917131 (+ 56us) tablet.cc:699] Row locks acquired
1214 20:52:25.917133 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:25.917159 (+ 26us) write_op.cc:270] Start()
1214 20:52:25.917182 (+ 23us) write_op.cc:276] Timestamp: P: 1765745545917098 usec, L: 0
1214 20:52:25.917185 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:25.917960 (+ 775us) log.cc:844] Serialized 433 byte log entry
1214 20:52:26.085921 (+167961us) op_driver.cc:464] REPLICATION: finished
1214 20:52:26.087313 (+ 1392us) write_op.cc:301] APPLY: starting
1214 20:52:26.087324 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:26.087366 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:26.087400 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:26.087510 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:26.087539 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:26.087546 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:26.087601 (+ 55us) log.cc:844] Serialized 29 byte log entry
1214 20:52:26.087641 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.087655 (+ 14us) write_op.cc:454] Released schema lock
1214 20:52:26.087661 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":5918976,"child_traces":[["op",{"apply.queue_time_us":1361,"apply.run_cpu_time_us":405,"apply.run_wall_time_us":405,"num_ops":2,"prepare.queue_time_us":18794,"prepare.run_cpu_time_us":559,"prepare.run_wall_time_us":1246,"raft.queue_time_us":43,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":168718,"threads_started":1}]]}
I20251214 20:52:26.090729 1202 tablet_bootstrap.cc:654] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:26.091053 449 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:50727 (request call id 190) took 861 ms (client timeout 1000 ms). Trace:
W20251214 20:52:26.091198 449 rpcz_store.cc:269] 1214 20:52:25.229570 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.294929 (+ 65359us) service_pool.cc:225] Handling call
1214 20:52:25.322761 (+ 27832us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b17aa08 00007ff411c87ca6 00007ff411c879d5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:26.091014 (+768253us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":8,"spinlock_wait_cycles":61715840}
I20251214 20:52:26.083016 962 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:26.094486 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d164747cb294bd4b126b4420231867d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:26.086378 1977 rpcz_store.cc:276] 1214 20:52:23.012524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.012568 (+ 44us) service_pool.cc:225] Handling call
1214 20:52:26.086233 (+3073665us) ts_tablet_manager.cc:1151] Deleting tablet f84e9dfc23d34eb0a379e8ac5a442e59
1214 20:52:26.086240 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.086306 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3067984}
I20251214 20:52:26.097641 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 208) took 3077 ms. Trace:
I20251214 20:52:26.099002 1977 rpcz_store.cc:276] 1214 20:52:23.020516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.020560 (+ 44us) service_pool.cc:225] Handling call
1214 20:52:26.097532 (+3076972us) ts_tablet_manager.cc:1151] Deleting tablet 9c0f49e5d78240a296e47eccb3218e4f
1214 20:52:26.097542 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.097623 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3076753}
I20251214 20:52:26.099957 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 209) took 3078 ms. Trace:
I20251214 20:52:26.101045 713 leader_election.cc:304] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:26.101438 1977 rpcz_store.cc:276] 1214 20:52:23.021088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.021215 (+ 127us) service_pool.cc:225] Handling call
1214 20:52:26.099845 (+3078630us) ts_tablet_manager.cc:1151] Deleting tablet 7fcfcfe81fe24d369eea696efc61d21c
1214 20:52:26.099855 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.099938 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3078480}
W20251214 20:52:26.101573 989 leader_election.cc:343] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.101704 989 leader_election.cc:304] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:26.102018 2455 raft_consensus.cc:2749] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:26.102145 989 leader_election.cc:343] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:26.102435 989 leader_election.cc:343] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.102484 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 210) took 3081 ms. Trace:
I20251214 20:52:26.102535 989 leader_election.cc:304] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:26.102876 989 leader_election.cc:343] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.103065 2455 raft_consensus.cc:2749] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:26.103153 989 leader_election.cc:343] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:26.103425 989 leader_election.cc:343] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:26.103704 989 consensus_peers.cc:597] T f900ebe92efa476cac232131a818d449 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:26.102545 1977 rpcz_store.cc:276] 1214 20:52:23.021517 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.021554 (+ 37us) service_pool.cc:225] Handling call
1214 20:52:26.102382 (+3080828us) ts_tablet_manager.cc:1151] Deleting tablet 52daf1204bf142e7b82852bdec6582fb
1214 20:52:26.102392 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.102468 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3080690}
I20251214 20:52:26.104362 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 211) took 3082 ms. Trace:
I20251214 20:52:26.104456 1977 rpcz_store.cc:276] 1214 20:52:23.021868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.021904 (+ 36us) service_pool.cc:225] Handling call
1214 20:52:26.104271 (+3082367us) ts_tablet_manager.cc:1151] Deleting tablet 472e4a5990e64d02b1d52dc6e86d7190
1214 20:52:26.104282 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.104341 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3082226}
I20251214 20:52:26.104702 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 212) took 3082 ms. Trace:
I20251214 20:52:26.104776 1977 rpcz_store.cc:276] 1214 20:52:23.022308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.022348 (+ 40us) service_pool.cc:225] Handling call
1214 20:52:26.104639 (+3082291us) ts_tablet_manager.cc:1151] Deleting tablet cd88d1e95ee04ef596863e1ae3687de0
1214 20:52:26.104646 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.104691 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3082156}
I20251214 20:52:26.105010 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 213) took 3082 ms. Trace:
I20251214 20:52:26.105072 1977 rpcz_store.cc:276] 1214 20:52:23.022661 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.022745 (+ 84us) service_pool.cc:225] Handling call
1214 20:52:26.104949 (+3082204us) ts_tablet_manager.cc:1151] Deleting tablet bb698ff30fcc47d09000cc68d2c5225b
1214 20:52:26.104956 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.104999 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3082063}
I20251214 20:52:26.105302 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 215) took 3070 ms. Trace:
I20251214 20:52:26.105401 1977 rpcz_store.cc:276] 1214 20:52:23.034571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.034611 (+ 40us) service_pool.cc:225] Handling call
1214 20:52:26.105241 (+3070630us) ts_tablet_manager.cc:1151] Deleting tablet a10663bd306d40e7a6375d06d69be22d
1214 20:52:26.105248 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.105290 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3070421}
I20251214 20:52:26.105716 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 216) took 3070 ms. Trace:
I20251214 20:52:26.105790 1977 rpcz_store.cc:276] 1214 20:52:23.035032 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.035277 (+ 245us) service_pool.cc:225] Handling call
1214 20:52:26.105613 (+3070336us) ts_tablet_manager.cc:1151] Deleting tablet 1dca369dc5f54a4ea1783e334c5670d6
1214 20:52:26.105621 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.105703 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3070142}
I20251214 20:52:26.106031 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 217) took 3060 ms. Trace:
I20251214 20:52:26.106105 1977 rpcz_store.cc:276] 1214 20:52:23.045145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.045207 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:26.105968 (+3060761us) ts_tablet_manager.cc:1151] Deleting tablet e16fe751f10f4616a00d0af811998729
1214 20:52:26.105976 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.106020 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3060516}
I20251214 20:52:26.106366 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 218) took 3060 ms. Trace:
I20251214 20:52:26.106451 1977 rpcz_store.cc:276] 1214 20:52:23.045655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.045788 (+ 133us) service_pool.cc:225] Handling call
1214 20:52:26.106283 (+3060495us) ts_tablet_manager.cc:1151] Deleting tablet ec85857f047e44d78835305ddcb70768
1214 20:52:26.106290 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.106354 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3060354}
I20251214 20:52:26.106747 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 219) took 3061 ms. Trace:
I20251214 20:52:26.106835 1977 rpcz_store.cc:276] 1214 20:52:23.045750 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.046038 (+ 288us) service_pool.cc:225] Handling call
1214 20:52:26.106656 (+3060618us) ts_tablet_manager.cc:1151] Deleting tablet 0c973e0111804b8b869068bdcc5869bf
1214 20:52:26.106664 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.106732 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3060494}
I20251214 20:52:26.107036 1977 tablet_replica.cc:333] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.107838 1856 tablet_replica.cc:333] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:26.108392 1856 raft_consensus.cc:2243] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.116592 2340 raft_consensus.cc:3060] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.119892 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ba50e32f7e724868bb891045a939e362 (table default.Table_8e3f7e1435ea493fa4729ca0d9e23573 [id=c3e5006d02fc4b69987feb30ba66d069]) successfully deleted
I20251214 20:52:26.120158 2340 raft_consensus.cc:2749] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:26.120788 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c9b7149a436a4cab80b39adccf8567f9 because the tablet was not found. No further retry: Not found: Tablet not found: c9b7149a436a4cab80b39adccf8567f9
W20251214 20:52:26.121227 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2842f1dc0c374d24946f90dac1cf8aa5 because the tablet was not found. No further retry: Not found: Tablet not found: 2842f1dc0c374d24946f90dac1cf8aa5
W20251214 20:52:26.121798 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3c3962c33a294fa98f8bd485268aee71 because the tablet was not found. No further retry: Not found: Tablet not found: 3c3962c33a294fa98f8bd485268aee71
W20251214 20:52:26.122190 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet bf8643c7e4094f529cd55656c7b3edb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf8643c7e4094f529cd55656c7b3edb6 already in progress: creating tablet
W20251214 20:52:26.122570 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 02e256d4537249a394b80dc36beefef4 because the tablet was not found. No further retry: Not found: Tablet not found: 02e256d4537249a394b80dc36beefef4
W20251214 20:52:26.122917 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet f84e9dfc23d34eb0a379e8ac5a442e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f84e9dfc23d34eb0a379e8ac5a442e59 already in progress: creating tablet
W20251214 20:52:26.123334 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 9c0f49e5d78240a296e47eccb3218e4f because the tablet was not found. No further retry: Not found: Tablet not found: 9c0f49e5d78240a296e47eccb3218e4f
W20251214 20:52:26.124044 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 7fcfcfe81fe24d369eea696efc61d21c because the tablet was not found. No further retry: Not found: Tablet not found: 7fcfcfe81fe24d369eea696efc61d21c
W20251214 20:52:26.124414 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 52daf1204bf142e7b82852bdec6582fb because the tablet was not found. No further retry: Not found: Tablet not found: 52daf1204bf142e7b82852bdec6582fb
I20251214 20:52:26.124747 1206 tablet_bootstrap.cc:492] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:26.124914 1206 ts_tablet_manager.cc:1403] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.241s user 0.004s sys 0.006s
W20251214 20:52:26.125370 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 472e4a5990e64d02b1d52dc6e86d7190 because the tablet was not found. No further retry: Not found: Tablet not found: 472e4a5990e64d02b1d52dc6e86d7190
W20251214 20:52:26.125713 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet cd88d1e95ee04ef596863e1ae3687de0 because the tablet was not found. No further retry: Not found: Tablet not found: cd88d1e95ee04ef596863e1ae3687de0
I20251214 20:52:26.126086 1206 raft_consensus.cc:359] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.126313 1206 raft_consensus.cc:385] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.126367 1206 raft_consensus.cc:740] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
W20251214 20:52:26.126420 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet bb698ff30fcc47d09000cc68d2c5225b because the tablet was not found. No further retry: Not found: Tablet not found: bb698ff30fcc47d09000cc68d2c5225b
I20251214 20:52:26.126569 1206 consensus_queue.cc:260] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:26.126839 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a10663bd306d40e7a6375d06d69be22d because the tablet was not found. No further retry: Not found: Tablet not found: a10663bd306d40e7a6375d06d69be22d
I20251214 20:52:26.127278 1206 ts_tablet_manager.cc:1434] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251214 20:52:26.127286 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 1dca369dc5f54a4ea1783e334c5670d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dca369dc5f54a4ea1783e334c5670d6 already in progress: creating tablet
I20251214 20:52:26.127627 1206 tablet_bootstrap.cc:492] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:26.127853 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet e16fe751f10f4616a00d0af811998729 because the tablet was not found. No further retry: Not found: Tablet not found: e16fe751f10f4616a00d0af811998729
I20251214 20:52:26.129743 2439 sys_catalog.cc:263] Verifying existing consensus state
W20251214 20:52:26.132560 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ec85857f047e44d78835305ddcb70768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec85857f047e44d78835305ddcb70768 already in progress: creating tablet
W20251214 20:52:26.133245 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 0c973e0111804b8b869068bdcc5869bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c973e0111804b8b869068bdcc5869bf already in progress: creating tablet
I20251214 20:52:26.129944 1103 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:26.138499 1856 raft_consensus.cc:2272] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.131953 2437 raft_consensus.cc:493] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:26.139446 2437 raft_consensus.cc:515] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.141017 2437 leader_election.cc:290] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:26.141139 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bdcecba5534294aca1eb419f18e9d2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:26.141927 991 leader_election.cc:343] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.147387 2437 raft_consensus.cc:493] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:26.147629 2437 raft_consensus.cc:515] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.148888 2437 leader_election.cc:290] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:26.149003 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a38e82f5dd1442f3a1dfcd6fbd94dbaf" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:26.149533 903 raft_consensus.cc:2468] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:26.150031 991 leader_election.cc:304] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:26.150530 2437 raft_consensus.cc:2804] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:26.150563 1203 tablet_bootstrap.cc:492] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:26.150868 1203 ts_tablet_manager.cc:1403] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.272s user 0.009s sys 0.001s
I20251214 20:52:26.151831 1203 raft_consensus.cc:359] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.152243 1203 raft_consensus.cc:385] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.152392 1203 raft_consensus.cc:740] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.152726 1203 consensus_queue.cc:260] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.153371 1203 ts_tablet_manager.cc:1434] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:26.156672 2439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Bootstrap starting.
W20251214 20:52:26.161310 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 56375.1us
I20251214 20:52:26.165966 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50088 (request call id 301) took 1019 ms. Trace:
I20251214 20:52:26.166206 452 rpcz_store.cc:276] 1214 20:52:25.146634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.915537 (+768903us) service_pool.cc:225] Handling call
1214 20:52:25.916501 (+ 964us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:25.916978 (+ 477us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:26.165655 (+248677us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:26.165802 (+ 147us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:26.165911 (+ 109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:25.944864 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:25.945553 (+ 689us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:25.945560 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:25.945563 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:25.945659 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:25.945670 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:25.945674 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:25.945765 (+ 91us) tablet.cc:699] Row locks acquired
1214 20:52:25.945767 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:25.945812 (+ 45us) write_op.cc:270] Start()
1214 20:52:25.945858 (+ 46us) write_op.cc:276] Timestamp: P: 1765745545945740 usec, L: 0
1214 20:52:25.945863 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:25.947600 (+ 1737us) log.cc:844] Serialized 433 byte log entry
1214 20:52:26.162383 (+214783us) op_driver.cc:464] REPLICATION: finished
1214 20:52:26.164997 (+ 2614us) write_op.cc:301] APPLY: starting
1214 20:52:26.165038 (+ 41us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:26.165102 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:26.165129 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:26.165274 (+ 145us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:26.165316 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:26.165323 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:26.165422 (+ 99us) log.cc:844] Serialized 29 byte log entry
1214 20:52:26.165490 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.165510 (+ 20us) write_op.cc:454] Released schema lock
1214 20:52:26.165516 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2567,"apply.run_cpu_time_us":595,"apply.run_wall_time_us":595,"num_ops":2,"prepare.queue_time_us":4104,"prepare.run_cpu_time_us":798,"prepare.run_wall_time_us":3632,"raft.queue_time_us":21,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":216503}]]}
I20251214 20:52:26.168946 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_27e3193bffa748be95f7c923662ad899" } new_table_name: "default.Table_27579b7155b140b3bdda5014c1eb57a8" modify_external_catalogs: true
I20251214 20:52:26.170646 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_41d7d906f2714c57832f29b1a2692574" } new_table_name: "default.Table_5de6830aaec645f2992e4cfaceb20c27" modify_external_catalogs: true
I20251214 20:52:26.172183 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_4fdc1b366f04459b8b083811e9740ae1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:26.172808 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fdc1b366f04459b8b083811e9740ae1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.174180 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ba7f3997d83f41b3a3a7019d61267c0e" } modify_external_catalogs: true
I20251214 20:52:26.174357 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ba7f3997d83f41b3a3a7019d61267c0e" } modify_external_catalogs: true
I20251214 20:52:26.175958 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9e2910a477ea4326b888fa02a5fa5a79" } modify_external_catalogs: true
I20251214 20:52:26.176152 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9e2910a477ea4326b888fa02a5fa5a79" } modify_external_catalogs: true
W20251214 20:52:26.177095 848 consensus_peers.cc:597] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35a529702f924a5c9bf53c3c668c6094. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:26.177794 1977 raft_consensus.cc:2243] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.178193 1977 raft_consensus.cc:2272] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.179034 1977 ts_tablet_manager.cc:1916] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.184929 1977 ts_tablet_manager.cc:1929] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.185153 1977 log.cc:1199] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/762068d560ff47918dd9fd8791083433
I20251214 20:52:26.185602 1977 ts_tablet_manager.cc:1950] T 762068d560ff47918dd9fd8791083433 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.185956 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 220) took 3139 ms. Trace:
I20251214 20:52:26.186105 1977 rpcz_store.cc:276] 1214 20:52:23.046338 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.046374 (+ 36us) service_pool.cc:225] Handling call
1214 20:52:26.106995 (+3060621us) ts_tablet_manager.cc:1151] Deleting tablet 762068d560ff47918dd9fd8791083433
1214 20:52:26.107002 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.178092 (+ 71090us) spinlock_profiling.cc:243] Waited 59.1 ms on lock 0x6160001a2c54. stack: 00007fa48881eee6 00007fa48881eca0 00007fa4877fa3ad 0000000000984f94 0000000000984f5d 000000000098492a 00007fa4947081e0 00007fa495a560b3 00007fa498616757 00007fa498649744 00007fa49862a240 00007fa498629fcd 0000000000a8970c 00007fa48886520f 00007fa48886852b 00007fa4888683a8
1214 20:52:26.182101 (+ 4009us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.184833 (+ 2732us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.185941 (+ 1108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":135946624,"tablet-delete.queue_time_us":3060476}
I20251214 20:52:26.186678 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 221) took 3140 ms. Trace:
I20251214 20:52:26.186789 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 762068d560ff47918dd9fd8791083433 (table default.Table_431bf74dfb32400bb0b2ef098c18e8df [id=2fcb81bfc6904fdd924de2494aba2e4a]) successfully deleted
I20251214 20:52:26.186810 1977 rpcz_store.cc:276] 1214 20:52:23.046680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.046732 (+ 52us) service_pool.cc:225] Handling call
1214 20:52:26.186563 (+3139831us) ts_tablet_manager.cc:1151] Deleting tablet dfe50a6219134431b0a025317932609b
1214 20:52:26.186572 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.186665 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3139705}
W20251214 20:52:26.187237 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet dfe50a6219134431b0a025317932609b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfe50a6219134431b0a025317932609b already in progress: creating tablet
I20251214 20:52:26.187604 1977 tablet_replica.cc:333] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.187808 1977 raft_consensus.cc:2243] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.187985 1977 raft_consensus.cc:2272] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.188825 1977 ts_tablet_manager.cc:1916] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:26.191226 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 75811.5us
I20251214 20:52:26.191857 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bdcecba5534294aca1eb419f18e9d2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:26.192459 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a38e82f5dd1442f3a1dfcd6fbd94dbaf" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:26.194262 1977 ts_tablet_manager.cc:1929] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.194370 1977 log.cc:1199] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/2accf8a160584c82bb695a52e3f2b05c
I20251214 20:52:26.194710 1977 ts_tablet_manager.cc:1950] T 2accf8a160584c82bb695a52e3f2b05c P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.194974 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 222) took 3147 ms. Trace:
I20251214 20:52:26.192428 1206 tablet_bootstrap.cc:654] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.195506 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:26.195881 1202 tablet_bootstrap.cc:492] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:26.196075 1202 ts_tablet_manager.cc:1403] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.107s user 0.009s sys 0.041s
I20251214 20:52:26.196789 1202 raft_consensus.cc:359] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.196996 1202 raft_consensus.cc:385] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.197082 1202 raft_consensus.cc:740] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.197285 1202 consensus_queue.cc:260] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.197480 2426 consensus_queue.cc:1048] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:26.197916 1202 ts_tablet_manager.cc:1434] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:26.198240 1202 tablet_bootstrap.cc:492] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:26.199589 1202 tablet_bootstrap.cc:654] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.195037 1977 rpcz_store.cc:276] 1214 20:52:23.047006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.047042 (+ 36us) service_pool.cc:225] Handling call
1214 20:52:26.187563 (+3140521us) ts_tablet_manager.cc:1151] Deleting tablet 2accf8a160584c82bb695a52e3f2b05c
1214 20:52:26.187571 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.191666 (+ 4095us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.194180 (+ 2514us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.194959 (+ 779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3140377}
I20251214 20:52:26.200723 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 2accf8a160584c82bb695a52e3f2b05c (table default.Table_b4334a1d99a54c2eabe2fb909563313c [id=d31b208e2e284575ad429878129a28f5]) successfully deleted
I20251214 20:52:26.202703 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 223) took 3155 ms. Trace:
I20251214 20:52:26.202837 1977 rpcz_store.cc:276] 1214 20:52:23.047363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.047398 (+ 35us) service_pool.cc:225] Handling call
1214 20:52:26.201048 (+3153650us) ts_tablet_manager.cc:1151] Deleting tablet 3d949abef1cb41a9905482d81a98d2c1
1214 20:52:26.201070 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.202681 (+ 1611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3153513}
W20251214 20:52:26.203516 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3d949abef1cb41a9905482d81a98d2c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d949abef1cb41a9905482d81a98d2c1 already in progress: creating tablet
I20251214 20:52:26.203994 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 224) took 3156 ms. Trace:
I20251214 20:52:26.204209 1977 rpcz_store.cc:276] 1214 20:52:23.047868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.047906 (+ 38us) service_pool.cc:225] Handling call
1214 20:52:26.203861 (+3155955us) ts_tablet_manager.cc:1151] Deleting tablet 6eca34627bbc4a3f9915be72f3128370
1214 20:52:26.203872 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.203960 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3155826}
W20251214 20:52:26.204854 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 6eca34627bbc4a3f9915be72f3128370 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6eca34627bbc4a3f9915be72f3128370 already in progress: creating tablet
I20251214 20:52:26.205267 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 225) took 3157 ms. Trace:
I20251214 20:52:26.205952 1977 rpcz_store.cc:276] 1214 20:52:23.048179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.048214 (+ 35us) service_pool.cc:225] Handling call
1214 20:52:26.205173 (+3156959us) ts_tablet_manager.cc:1151] Deleting tablet 4d6e7306254044449f1127fb963ca5f0
1214 20:52:26.205181 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.205253 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3156840}
W20251214 20:52:26.206547 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 4d6e7306254044449f1127fb963ca5f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d6e7306254044449f1127fb963ca5f0 already in progress: creating tablet
I20251214 20:52:26.207150 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 226) took 3158 ms. Trace:
I20251214 20:52:26.207512 1977 rpcz_store.cc:276] 1214 20:52:23.048491 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.048529 (+ 38us) service_pool.cc:225] Handling call
1214 20:52:26.207030 (+3158501us) ts_tablet_manager.cc:1151] Deleting tablet b1f4ea72c6274569a98cab6afdc55d55
1214 20:52:26.207038 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.207111 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3158379}
W20251214 20:52:26.208060 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet b1f4ea72c6274569a98cab6afdc55d55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1f4ea72c6274569a98cab6afdc55d55 already in progress: creating tablet
I20251214 20:52:26.208438 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 227) took 3159 ms. Trace:
I20251214 20:52:26.208493 2426 consensus_queue.cc:1048] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:26.209306 1977 rpcz_store.cc:276] 1214 20:52:23.048910 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.053158 (+ 4248us) service_pool.cc:225] Handling call
1214 20:52:26.208349 (+3155191us) ts_tablet_manager.cc:1151] Deleting tablet 292a28b12df246a8bc855d81c530746e
1214 20:52:26.208357 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.208424 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3155038}
W20251214 20:52:26.209952 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 292a28b12df246a8bc855d81c530746e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 292a28b12df246a8bc855d81c530746e already in progress: creating tablet
W20251214 20:52:26.210031 448 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed9790fbebda4f848291930bf1dd8174 has no consensus state
I20251214 20:52:26.210328 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 228) took 3156 ms. Trace:
I20251214 20:52:26.210656 1977 rpcz_store.cc:276] 1214 20:52:23.054002 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.055273 (+ 1271us) service_pool.cc:225] Handling call
1214 20:52:26.210239 (+3154966us) ts_tablet_manager.cc:1151] Deleting tablet c8e33ba1149a44c294885d76f676160a
1214 20:52:26.210247 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.210315 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3154761}
W20251214 20:52:26.211243 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c8e33ba1149a44c294885d76f676160a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8e33ba1149a44c294885d76f676160a already in progress: creating tablet
I20251214 20:52:26.211673 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 229) took 3155 ms. Trace:
I20251214 20:52:26.212028 1977 rpcz_store.cc:276] 1214 20:52:23.055783 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.055960 (+ 177us) service_pool.cc:225] Handling call
1214 20:52:26.211571 (+3155611us) ts_tablet_manager.cc:1151] Deleting tablet 8eb1006b0ba440dcba2f9de04dca2702
1214 20:52:26.211580 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.211656 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3155460}
W20251214 20:52:26.212679 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8eb1006b0ba440dcba2f9de04dca2702 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8eb1006b0ba440dcba2f9de04dca2702 already in progress: creating tablet
I20251214 20:52:26.213068 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 230) took 3156 ms. Trace:
I20251214 20:52:26.213415 1977 rpcz_store.cc:276] 1214 20:52:23.056582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.056663 (+ 81us) service_pool.cc:225] Handling call
1214 20:52:26.212964 (+3156301us) ts_tablet_manager.cc:1151] Deleting tablet 0d7c7b3f62764ff497f4d70b53eed7de
1214 20:52:26.212973 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.213051 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3156083}
W20251214 20:52:26.214032 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 0d7c7b3f62764ff497f4d70b53eed7de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d7c7b3f62764ff497f4d70b53eed7de already in progress: creating tablet
I20251214 20:52:26.214413 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 231) took 3157 ms. Trace:
I20251214 20:52:26.214756 1977 rpcz_store.cc:276] 1214 20:52:23.057064 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.057224 (+ 160us) service_pool.cc:225] Handling call
1214 20:52:26.214319 (+3157095us) ts_tablet_manager.cc:1151] Deleting tablet 47e813373fbd482298312ead4f037c01
1214 20:52:26.214327 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.214397 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3156928}
W20251214 20:52:26.215538 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 47e813373fbd482298312ead4f037c01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e813373fbd482298312ead4f037c01 already in progress: creating tablet
I20251214 20:52:26.215922 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 232) took 3158 ms. Trace:
I20251214 20:52:26.216234 1977 rpcz_store.cc:276] 1214 20:52:23.057577 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.059384 (+ 1807us) service_pool.cc:225] Handling call
1214 20:52:26.215821 (+3156437us) ts_tablet_manager.cc:1151] Deleting tablet 17550eb0ee6b476f8a8c304451f678b2
1214 20:52:26.215830 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.215905 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3156253}
W20251214 20:52:26.217332 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 17550eb0ee6b476f8a8c304451f678b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17550eb0ee6b476f8a8c304451f678b2 already in progress: creating tablet
I20251214 20:52:26.217015 1202 tablet_bootstrap.cc:492] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:26.217694 1202 ts_tablet_manager.cc:1403] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20251214 20:52:26.218485 1202 raft_consensus.cc:359] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.218734 1202 raft_consensus.cc:385] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.218915 1202 raft_consensus.cc:740] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.219223 1202 consensus_queue.cc:260] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.221782 1202 ts_tablet_manager.cc:1434] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251214 20:52:26.222082 1202 tablet_bootstrap.cc:492] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:26.223615 1202 tablet_bootstrap.cc:654] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.223822 1977 tablet_replica.cc:333] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.224614 1977 raft_consensus.cc:2243] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.224805 1977 raft_consensus.cc:2272] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.225322 1977 ts_tablet_manager.cc:1916] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.228917 1203 tablet_bootstrap.cc:492] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:26.231117 1203 tablet_bootstrap.cc:654] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.231186 1977 ts_tablet_manager.cc:1929] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.233893 1977 log.cc:1199] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ee7d709316b24b8eb223a838710c2979
I20251214 20:52:26.234422 1977 ts_tablet_manager.cc:1950] T ee7d709316b24b8eb223a838710c2979 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.237704 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 233) took 3175 ms. Trace:
I20251214 20:52:26.238021 1977 rpcz_store.cc:276] 1214 20:52:23.100593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.109158 (+ 8565us) service_pool.cc:225] Handling call
1214 20:52:26.223783 (+3114625us) ts_tablet_manager.cc:1151] Deleting tablet ee7d709316b24b8eb223a838710c2979
1214 20:52:26.223791 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.228339 (+ 4548us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.231082 (+ 2743us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.237683 (+ 6601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3114358}
I20251214 20:52:26.238934 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ee7d709316b24b8eb223a838710c2979 (table default.Table_aa507956e1b54af68730c20f7c10e842 [id=f249932331aa4016806e482e8c7c4179]) successfully deleted
I20251214 20:52:26.239526 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 236) took 3067 ms. Trace:
I20251214 20:52:26.239965 1977 rpcz_store.cc:276] 1214 20:52:23.171715 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.171773 (+ 58us) service_pool.cc:225] Handling call
1214 20:52:26.239380 (+3067607us) ts_tablet_manager.cc:1151] Deleting tablet 65e79db74fa046698507c83416ee1669
1214 20:52:26.239392 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.239508 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2911104}
W20251214 20:52:26.240672 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 65e79db74fa046698507c83416ee1669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e79db74fa046698507c83416ee1669 already in progress: creating tablet
I20251214 20:52:26.241020 1977 tablet_replica.cc:333] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.241824 1977 raft_consensus.cc:2243] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.242189 1977 raft_consensus.cc:2272] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.243057 1977 ts_tablet_manager.cc:1916] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:26.251423 440 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet ed9790fbebda4f848291930bf1dd8174 (table default.Table_0945dd902049447298aef35db32f33ba [id=3b87708d1c7b460d925f7da030a1a23f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ed9790fbebda4f848291930bf1dd8174 has no consensus state
I20251214 20:52:26.251389 1982 ts_tablet_manager.cc:1929] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:26.251544 1982 log.cc:1199] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/604b7c6fe3b24daca70711a1990af556
I20251214 20:52:26.251994 1982 ts_tablet_manager.cc:1950] T 604b7c6fe3b24daca70711a1990af556 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:26.252493 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 210) took 3923 ms. Trace:
I20251214 20:52:26.252640 1982 rpcz_store.cc:276] 1214 20:52:22.329418 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.329474 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:25.933337 (+3603863us) ts_tablet_manager.cc:1151] Deleting tablet 604b7c6fe3b24daca70711a1990af556
1214 20:52:25.933349 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.070582 (+137233us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.251236 (+180654us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.252472 (+ 1236us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3603658}
I20251214 20:52:26.253165 1982 tablet_replica.cc:333] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:26.253368 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 604b7c6fe3b24daca70711a1990af556 (table default.Table_fbb5b21626c142c8baac60a7f29d3c51 [id=7cb78d704474480db14d212ee89166a0]) successfully deleted
I20251214 20:52:26.256402 1977 ts_tablet_manager.cc:1929] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:26.262220 1977 log.cc:1199] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/242a77ae57174de48f4eaf43384ee0ba
I20251214 20:52:26.262774 1977 ts_tablet_manager.cc:1950] T 242a77ae57174de48f4eaf43384ee0ba P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.263942 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 237) took 3091 ms. Trace:
I20251214 20:52:26.264925 1977 rpcz_store.cc:276] 1214 20:52:23.172112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.172178 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:26.240976 (+3068798us) ts_tablet_manager.cc:1151] Deleting tablet 242a77ae57174de48f4eaf43384ee0ba
1214 20:52:26.240985 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.251182 (+ 10197us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.256285 (+ 5103us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.263921 (+ 7636us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2912499}
I20251214 20:52:26.150686 2437 raft_consensus.cc:493] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251214 20:52:26.194006 989 leader_election.cc:343] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.265748 989 leader_election.cc:304] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:26.265931 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 242a77ae57174de48f4eaf43384ee0ba (table default.Table_9ad469dea994477983bfe62eef37e334 [id=b1be575d16354425ab4f7e8e62847820]) successfully deleted
W20251214 20:52:26.266156 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 72155.8us
W20251214 20:52:26.266556 989 leader_election.cc:343] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.267014 2431 raft_consensus.cc:2749] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:26.155193 1856 ts_tablet_manager.cc:1916] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:26.245033 991 consensus_peers.cc:597] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2accf8a160584c82bb695a52e3f2b05c. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:26.272439 1977 tablet_replica.cc:333] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.272662 1977 raft_consensus.cc:2243] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.272809 1977 raft_consensus.cc:2272] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.273326 1977 ts_tablet_manager.cc:1916] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.267030 2437 raft_consensus.cc:3060] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.276429 2437 raft_consensus.cc:515] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:26.285306 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bea3e9d7334345fa88ededf1be17a434 consensus state has no leader
I20251214 20:52:26.285728 446 master_service.cc:954] ReplaceTablet: received request to replace tablet e7b247b761c94c60ad3a468db371b30b from {username='slave'} at 127.0.0.1:50088
W20251214 20:52:26.287369 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6bdfbe4d15345de8be5de00bfe4ca07 consensus state has no leader
I20251214 20:52:26.287773 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_cb1220aeec334a3c8aa15d0049c58d75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:26.288492 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb1220aeec334a3c8aa15d0049c58d75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.279070 1977 ts_tablet_manager.cc:1929] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:26.295918 1977 log.cc:1199] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/b2e83bca8ed642b6a1b705ef70fde27f
I20251214 20:52:26.296518 1977 ts_tablet_manager.cc:1950] T b2e83bca8ed642b6a1b705ef70fde27f P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.297695 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 238) took 3123 ms. Trace:
I20251214 20:52:26.298118 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet e59b80ee919b4e1393fe317ed0e3a3d2
I20251214 20:52:26.298795 1977 rpcz_store.cc:276] 1214 20:52:23.173826 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.173876 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:26.271718 (+3097842us) ts_tablet_manager.cc:1151] Deleting tablet b2e83bca8ed642b6a1b705ef70fde27f
1214 20:52:26.271732 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.276334 (+ 4602us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.278994 (+ 2660us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.297674 (+ 18680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2943138}
I20251214 20:52:26.299846 878 tablet_service.cc:1552] Processing DeleteTablet for tablet e59b80ee919b4e1393fe317ed0e3a3d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.300150 1977 tablet_replica.cc:333] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.300410 1977 raft_consensus.cc:2243] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.300460 2437 leader_election.cc:290] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:26.300951 1977 raft_consensus.cc:2272] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.300875 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a38e82f5dd1442f3a1dfcd6fbd94dbaf" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:26.301482 904 raft_consensus.cc:3060] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.304036 904 raft_consensus.cc:2468] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:26.304173 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet e59b80ee919b4e1393fe317ed0e3a3d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.304702 991 leader_election.cc:304] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:26.305331 2364 raft_consensus.cc:2804] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:26.305644 2364 raft_consensus.cc:697] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:26.305634 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_17954423e5674f0eb5a9b53bc2e75913"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:26.306438 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17954423e5674f0eb5a9b53bc2e75913 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.306891 2364 consensus_queue.cc:237] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.311507 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet b2e83bca8ed642b6a1b705ef70fde27f (table default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7]) successfully deleted
I20251214 20:52:26.277841 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a38e82f5dd1442f3a1dfcd6fbd94dbaf" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:26.312196 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_94c723f6fee24f40a5787f7e5e92d03f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:26.312912 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94c723f6fee24f40a5787f7e5e92d03f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:26.312995 851 consensus_peers.cc:597] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:26.312999 989 leader_election.cc:343] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.313832 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet d745583012b4465090d85c75470e0bc2
W20251214 20:52:26.314877 989 consensus_peers.cc:597] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:26.283053 2478 tablet.cc:1722] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:26.304126 746 tablet_service.cc:1552] Processing DeleteTablet for tablet e59b80ee919b4e1393fe317ed0e3a3d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.315414 1977 ts_tablet_manager.cc:1916] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.316265 1982 raft_consensus.cc:2243] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.316288 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_6eb267b64a6a485bb607876002c23ec4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:26.316560 1982 raft_consensus.cc:2272] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:26.316969 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eb267b64a6a485bb607876002c23ec4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.317157 1982 ts_tablet_manager.cc:1916] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.319293 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet d745583012b4465090d85c75470e0bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.321264 1856 ts_tablet_manager.cc:1929] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.323803 1856 log.cc:1199] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/7dd657f3baff40bb8cb7f4648367f7a1
I20251214 20:52:26.324322 1856 ts_tablet_manager.cc:1950] T 7dd657f3baff40bb8cb7f4648367f7a1 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.325788 1203 tablet_bootstrap.cc:492] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:26.326459 1203 ts_tablet_manager.cc:1403] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.000s
I20251214 20:52:26.327425 1203 raft_consensus.cc:359] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.327705 1203 raft_consensus.cc:385] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.327803 1203 raft_consensus.cc:740] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.318753 878 tablet_service.cc:1552] Processing DeleteTablet for tablet d745583012b4465090d85c75470e0bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.328109 1203 consensus_queue.cc:260] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.328585 1203 ts_tablet_manager.cc:1434] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:26.333832 1977 ts_tablet_manager.cc:1929] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.334149 746 tablet_service.cc:1552] Processing DeleteTablet for tablet d745583012b4465090d85c75470e0bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.336450 1203 tablet_bootstrap.cc:492] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:26.337361 2287 raft_consensus.cc:493] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:26.339403 2287 raft_consensus.cc:515] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.340118 2287 leader_election.cc:290] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:26.341354 851 leader_election.cc:343] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: bf8643c7e4094f529cd55656c7b3edb6
I20251214 20:52:26.342629 1203 tablet_bootstrap.cc:654] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.342530 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf8643c7e4094f529cd55656c7b3edb6" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:26.345978 848 leader_election.cc:343] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.346097 848 leader_election.cc:304] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:26.346635 2287 raft_consensus.cc:2749] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:26.350693 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 230) took 4194 ms. Trace:
I20251214 20:52:26.355660 1856 rpcz_store.cc:276] 1214 20:52:22.156522 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.156560 (+ 38us) service_pool.cc:225] Handling call
1214 20:52:26.107769 (+3951209us) ts_tablet_manager.cc:1151] Deleting tablet 7dd657f3baff40bb8cb7f4648367f7a1
1214 20:52:26.107785 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.280953 (+173168us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.321077 (+ 40124us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.350658 (+ 29581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3951062}
I20251214 20:52:26.356235 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 231) took 4188 ms. Trace:
I20251214 20:52:26.356415 1856 rpcz_store.cc:276] 1214 20:52:22.167373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.167430 (+ 57us) service_pool.cc:225] Handling call
1214 20:52:26.356110 (+4188680us) ts_tablet_manager.cc:1151] Deleting tablet 9d89ee602c524c96a21c0a235af07481
1214 20:52:26.356121 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.356218 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4188477}
I20251214 20:52:26.356438 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 7dd657f3baff40bb8cb7f4648367f7a1 (table default.Table_8b86c42f7826440982c88a8625559146 [id=c35009dabd4a42598506e52b3d3e493c]) successfully deleted
I20251214 20:52:26.356936 1856 tablet_replica.cc:333] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:26.356899 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 9d89ee602c524c96a21c0a235af07481 because the tablet was not found. No further retry: Not found: Tablet not found: 9d89ee602c524c96a21c0a235af07481
I20251214 20:52:26.357228 1856 raft_consensus.cc:2243] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.357877 1856 raft_consensus.cc:2272] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.358565 1856 ts_tablet_manager.cc:1916] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.358552 1977 log.cc:1199] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/35a529702f924a5c9bf53c3c668c6094
I20251214 20:52:26.359935 1977 ts_tablet_manager.cc:1950] T 35a529702f924a5c9bf53c3c668c6094 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.363938 446 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet e7b247b761c94c60ad3a468db371b30b
I20251214 20:52:26.364495 446 catalog_manager.cc:6724] ReplaceTablet: tablet e7b247b761c94c60ad3a468db371b30b deleted and replaced by tablet 2f75657d7bc44f6393c55ab119eb7dc2
I20251214 20:52:26.364916 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_1e4dafc4ff8940ea90956623c915e3fd" } new_table_name: "default.Table_6cfa43d70d2f426fb58f8ef32476c3ac" modify_external_catalogs: true
I20251214 20:52:26.367594 878 tablet_service.cc:1552] Processing DeleteTablet for tablet e7b247b761c94c60ad3a468db371b30b with delete_type TABLET_DATA_DELETED (replaced by tablet 2f75657d7bc44f6393c55ab119eb7dc2) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.367882 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 240) took 3183 ms. Trace:
I20251214 20:52:26.367993 1977 rpcz_store.cc:276] 1214 20:52:23.184369 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.184417 (+ 48us) service_pool.cc:225] Handling call
1214 20:52:26.299504 (+3115087us) ts_tablet_manager.cc:1151] Deleting tablet 35a529702f924a5c9bf53c3c668c6094
1214 20:52:26.299524 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.331032 (+ 31508us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.333738 (+ 2706us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.367856 (+ 34118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2967577}
I20251214 20:52:26.368134 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet e7b247b761c94c60ad3a468db371b30b with delete_type TABLET_DATA_DELETED (replaced by tablet 2f75657d7bc44f6393c55ab119eb7dc2) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.368538 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 35a529702f924a5c9bf53c3c668c6094 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]) successfully deleted
I20251214 20:52:26.370324 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_cd43be14c9f84889bd6e03167dd35f6c" } modify_external_catalogs: true
I20251214 20:52:26.370549 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_cd43be14c9f84889bd6e03167dd35f6c" } modify_external_catalogs: true
I20251214 20:52:26.372334 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_89dffab0bf1d4856b671651a07b3439e" } modify_external_catalogs: true
I20251214 20:52:26.372527 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_89dffab0bf1d4856b671651a07b3439e" } modify_external_catalogs: true
I20251214 20:52:26.372634 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 241) took 3188 ms. Trace:
I20251214 20:52:26.372730 1977 rpcz_store.cc:276] 1214 20:52:23.184626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.184672 (+ 46us) service_pool.cc:225] Handling call
1214 20:52:26.372474 (+3187802us) ts_tablet_manager.cc:1151] Deleting tablet 69f6108792fa4c2a9744e3248c76251f
1214 20:52:26.372487 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.372612 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3037100}
W20251214 20:52:26.373240 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 69f6108792fa4c2a9744e3248c76251f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f6108792fa4c2a9744e3248c76251f already in progress: creating tablet
I20251214 20:52:26.374111 1977 tablet_replica.cc:333] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.374644 1977 raft_consensus.cc:2243] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.375173 1977 raft_consensus.cc:2272] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.375996 1977 ts_tablet_manager.cc:1916] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.384444 746 tablet_service.cc:1552] Processing DeleteTablet for tablet e7b247b761c94c60ad3a468db371b30b with delete_type TABLET_DATA_DELETED (replaced by tablet 2f75657d7bc44f6393c55ab119eb7dc2) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.385354 1977 ts_tablet_manager.cc:1929] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.388607 1206 tablet_bootstrap.cc:492] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:26.388896 1206 ts_tablet_manager.cc:1403] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.261s user 0.020s sys 0.024s
I20251214 20:52:26.390547 1977 log.cc:1199] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/9d89ee602c524c96a21c0a235af07481
I20251214 20:52:26.390439 1206 raft_consensus.cc:359] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.390750 1206 raft_consensus.cc:385] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.390836 1206 raft_consensus.cc:740] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.391326 1977 ts_tablet_manager.cc:1950] T 9d89ee602c524c96a21c0a235af07481 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.392469 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 243) took 3207 ms. Trace:
I20251214 20:52:26.393450 1977 rpcz_store.cc:276] 1214 20:52:23.185079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.185127 (+ 48us) service_pool.cc:225] Handling call
1214 20:52:26.374057 (+3188930us) ts_tablet_manager.cc:1151] Deleting tablet 9d89ee602c524c96a21c0a235af07481
1214 20:52:26.374069 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.382415 (+ 8346us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.385240 (+ 2825us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.392449 (+ 7209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3034832}
I20251214 20:52:26.394320 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 9d89ee602c524c96a21c0a235af07481 (table default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]) successfully deleted
I20251214 20:52:26.393402 1206 consensus_queue.cc:260] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.394940 1977 tablet_replica.cc:333] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.395406 1977 raft_consensus.cc:2243] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.395607 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_47862fea18504c61af1783e5d8fca95e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:26.396239 1977 raft_consensus.cc:2272] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:26.396462 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47862fea18504c61af1783e5d8fca95e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.396710 1206 ts_tablet_manager.cc:1434] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20251214 20:52:26.398730 1206 tablet_bootstrap.cc:492] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:26.401484 1856 ts_tablet_manager.cc:1929] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.402402 1856 log.cc:1199] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/35a529702f924a5c9bf53c3c668c6094
I20251214 20:52:26.405356 1977 ts_tablet_manager.cc:1916] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.407047 1856 ts_tablet_manager.cc:1950] T 35a529702f924a5c9bf53c3c668c6094 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.394676 1982 ts_tablet_manager.cc:1929] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.407999 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 232) took 4240 ms. Trace:
I20251214 20:52:26.408037 1982 log.cc:1199] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/5da9deeedce0421c9aeaebd71ba11b1b
I20251214 20:52:26.411317 1982 ts_tablet_manager.cc:1950] T 5da9deeedce0421c9aeaebd71ba11b1b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:26.413219 1206 tablet_bootstrap.cc:654] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.408114 1856 rpcz_store.cc:276] 1214 20:52:22.167830 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.167873 (+ 43us) service_pool.cc:225] Handling call
1214 20:52:26.356883 (+4189010us) ts_tablet_manager.cc:1151] Deleting tablet 35a529702f924a5c9bf53c3c668c6094
1214 20:52:26.356892 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.376587 (+ 19695us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.397454 (+ 20867us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.407973 (+ 10519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4188846}
I20251214 20:52:26.418169 1977 ts_tablet_manager.cc:1929] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.418287 1856 tablet_replica.cc:333] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:26.418546 1856 raft_consensus.cc:2243] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.418737 1856 raft_consensus.cc:2272] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.419438 1856 ts_tablet_manager.cc:1916] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.419771 1977 log.cc:1199] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/5da9deeedce0421c9aeaebd71ba11b1b
I20251214 20:52:26.419740 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 35a529702f924a5c9bf53c3c668c6094 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]) successfully deleted
I20251214 20:52:26.420511 1977 ts_tablet_manager.cc:1950] T 5da9deeedce0421c9aeaebd71ba11b1b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.421851 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 242) took 3236 ms. Trace:
I20251214 20:52:26.422909 1977 rpcz_store.cc:276] 1214 20:52:23.184984 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.186229 (+ 1245us) service_pool.cc:225] Handling call
1214 20:52:26.394885 (+3208656us) ts_tablet_manager.cc:1151] Deleting tablet 5da9deeedce0421c9aeaebd71ba11b1b
1214 20:52:26.394897 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.411208 (+ 16311us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.418046 (+ 6838us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.421826 (+ 3780us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3055594}
I20251214 20:52:26.422993 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 207) took 4093 ms. Trace:
I20251214 20:52:26.423871 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 5da9deeedce0421c9aeaebd71ba11b1b (table default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]) successfully deleted
I20251214 20:52:26.424377 1977 tablet_replica.cc:333] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.424837 1977 raft_consensus.cc:2243] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.423830 1982 rpcz_store.cc:276] 1214 20:52:22.329119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.329497 (+ 378us) service_pool.cc:225] Handling call
1214 20:52:26.253109 (+3923612us) ts_tablet_manager.cc:1151] Deleting tablet 5da9deeedce0421c9aeaebd71ba11b1b
1214 20:52:26.253120 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.359917 (+106797us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.394538 (+ 34621us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.422739 (+ 28201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3923357}
I20251214 20:52:26.425561 1977 raft_consensus.cc:2272] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.426580 1977 ts_tablet_manager.cc:1916] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.428009 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 211) took 4098 ms. Trace:
I20251214 20:52:26.428562 1982 rpcz_store.cc:276] 1214 20:52:22.329564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.329648 (+ 84us) service_pool.cc:225] Handling call
1214 20:52:26.426139 (+4096491us) ts_tablet_manager.cc:1151] Deleting tablet f84e9dfc23d34eb0a379e8ac5a442e59
1214 20:52:26.426154 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.427987 (+ 1833us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4096204}
I20251214 20:52:26.429567 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 5da9deeedce0421c9aeaebd71ba11b1b (table default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]) successfully deleted
W20251214 20:52:26.430011 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f84e9dfc23d34eb0a379e8ac5a442e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f84e9dfc23d34eb0a379e8ac5a442e59 already in progress: creating tablet
I20251214 20:52:26.430902 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 206) took 4101 ms. Trace:
I20251214 20:52:26.431571 1982 rpcz_store.cc:276] 1214 20:52:22.329029 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.329617 (+ 588us) service_pool.cc:225] Handling call
1214 20:52:26.430296 (+4100679us) ts_tablet_manager.cc:1151] Deleting tablet 69f6108792fa4c2a9744e3248c76251f
1214 20:52:26.430309 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.430881 (+ 572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4100329}
W20251214 20:52:26.433020 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 69f6108792fa4c2a9744e3248c76251f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f6108792fa4c2a9744e3248c76251f already in progress: creating tablet
I20251214 20:52:26.433554 1982 tablet_replica.cc:333] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:26.433998 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_5126357c033645d1ae06a909124ab640"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:26.434659 1856 ts_tablet_manager.cc:1929] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.434775 1856 log.cc:1199] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/5da9deeedce0421c9aeaebd71ba11b1b
W20251214 20:52:26.434815 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5126357c033645d1ae06a909124ab640 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.435174 1856 ts_tablet_manager.cc:1950] T 5da9deeedce0421c9aeaebd71ba11b1b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.434998 1982 raft_consensus.cc:2243] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.436050 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 233) took 4267 ms. Trace:
I20251214 20:52:26.436138 1856 rpcz_store.cc:276] 1214 20:52:22.168227 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.168284 (+ 57us) service_pool.cc:225] Handling call
1214 20:52:26.418213 (+4249929us) ts_tablet_manager.cc:1151] Deleting tablet 5da9deeedce0421c9aeaebd71ba11b1b
1214 20:52:26.418229 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.424159 (+ 5930us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.434515 (+ 10356us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.436025 (+ 1510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4249802}
I20251214 20:52:26.438086 1982 raft_consensus.cc:2272] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.442075 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 5da9deeedce0421c9aeaebd71ba11b1b (table default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]) successfully deleted
I20251214 20:52:26.442585 1856 tablet_replica.cc:333] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:26.443003 2327 raft_consensus.cc:493] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:26.443152 1856 raft_consensus.cc:2243] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.443354 2327 raft_consensus.cc:515] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.443863 1856 raft_consensus.cc:2272] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.444098 1982 ts_tablet_manager.cc:1916] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.444495 1856 ts_tablet_manager.cc:1916] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.444823 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f198ef4dfa4f5791144b9a24fe693f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:26.444923 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ec7824f5eff4cc59c45acd3b3b3d194 consensus state has no leader
I20251214 20:52:26.445037 1050 raft_consensus.cc:2468] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:26.445688 851 leader_election.cc:304] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:26.445628 1206 tablet_bootstrap.cc:492] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:26.445993 2287 raft_consensus.cc:2804] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:26.445978 1206 ts_tablet_manager.cc:1403] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.001s
I20251214 20:52:26.446061 2287 raft_consensus.cc:493] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:26.446139 2287 raft_consensus.cc:3060] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.447531 1206 raft_consensus.cc:359] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.447835 1206 raft_consensus.cc:385] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.447921 1206 raft_consensus.cc:740] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.448297 1206 consensus_queue.cc:260] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.448904 2287 raft_consensus.cc:515] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.450167 2287 leader_election.cc:290] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:26.449934 1206 ts_tablet_manager.cc:1434] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:26.452303 1206 tablet_bootstrap.cc:492] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:26.455550 1203 tablet_bootstrap.cc:492] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:26.455717 1203 ts_tablet_manager.cc:1403] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.119s user 0.012s sys 0.012s
I20251214 20:52:26.456089 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f198ef4dfa4f5791144b9a24fe693f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:26.456297 1050 raft_consensus.cc:3060] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.456516 1203 raft_consensus.cc:359] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.456678 1203 raft_consensus.cc:385] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.456723 1203 raft_consensus.cc:740] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.456915 1203 consensus_queue.cc:260] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.458892 1050 raft_consensus.cc:2468] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:26.460104 851 leader_election.cc:304] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:26.464491 1203 ts_tablet_manager.cc:1434] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:26.465330 2287 raft_consensus.cc:2804] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:26.465694 2287 raft_consensus.cc:697] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:26.466017 2287 consensus_queue.cc:237] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.467412 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f198ef4dfa4f5791144b9a24fe693f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:26.468437 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d9f198ef4dfa4f5791144b9a24fe693f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:26.468547 848 leader_election.cc:343] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:26.469080 848 leader_election.cc:343] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.468835 2327 leader_election.cc:290] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:26.469640 1203 tablet_bootstrap.cc:492] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:26.471323 1202 tablet_bootstrap.cc:492] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:26.471787 1206 tablet_bootstrap.cc:654] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.472258 1202 ts_tablet_manager.cc:1403] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.250s user 0.025s sys 0.034s
I20251214 20:52:26.472478 1203 tablet_bootstrap.cc:654] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.473399 1202 raft_consensus.cc:359] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.473820 1856 ts_tablet_manager.cc:1929] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.473984 1856 log.cc:1199] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/2accf8a160584c82bb695a52e3f2b05c
I20251214 20:52:26.474426 1856 ts_tablet_manager.cc:1950] T 2accf8a160584c82bb695a52e3f2b05c P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.474637 1202 raft_consensus.cc:385] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.474794 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 234) took 4306 ms. Trace:
I20251214 20:52:26.474902 1202 raft_consensus.cc:740] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.474952 1856 rpcz_store.cc:276] 1214 20:52:22.168776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.168828 (+ 52us) service_pool.cc:225] Handling call
1214 20:52:26.442519 (+4273691us) ts_tablet_manager.cc:1151] Deleting tablet 2accf8a160584c82bb695a52e3f2b05c
1214 20:52:26.442532 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.451657 (+ 9125us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.473686 (+ 22029us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.474774 (+ 1088us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4273553}
I20251214 20:52:26.475646 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 2accf8a160584c82bb695a52e3f2b05c (table default.Table_b4334a1d99a54c2eabe2fb909563313c [id=d31b208e2e284575ad429878129a28f5]) successfully deleted
I20251214 20:52:26.475739 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 235) took 4297 ms. Trace:
I20251214 20:52:26.475808 1856 rpcz_store.cc:276] 1214 20:52:22.178079 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.178130 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:26.475600 (+4297470us) ts_tablet_manager.cc:1151] Deleting tablet 65e79db74fa046698507c83416ee1669
1214 20:52:26.475610 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.475723 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4297303}
I20251214 20:52:26.475409 1202 consensus_queue.cc:260] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:26.476270 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 65e79db74fa046698507c83416ee1669 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e79db74fa046698507c83416ee1669 already in progress: creating tablet
I20251214 20:52:26.476742 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 237) took 4297 ms. Trace:
I20251214 20:52:26.476843 1856 rpcz_store.cc:276] 1214 20:52:22.179076 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.179116 (+ 40us) service_pool.cc:225] Handling call
1214 20:52:26.476642 (+4297526us) ts_tablet_manager.cc:1151] Deleting tablet b2e83bca8ed642b6a1b705ef70fde27f
1214 20:52:26.476651 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.476725 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4297365}
W20251214 20:52:26.477234 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b2e83bca8ed642b6a1b705ef70fde27f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2e83bca8ed642b6a1b705ef70fde27f already in progress: creating tablet
I20251214 20:52:26.477310 1202 ts_tablet_manager.cc:1434] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:26.477632 1202 tablet_bootstrap.cc:492] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:26.478082 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 238) took 4298 ms. Trace:
I20251214 20:52:26.478197 1856 rpcz_store.cc:276] 1214 20:52:22.179741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.179800 (+ 59us) service_pool.cc:225] Handling call
1214 20:52:26.477986 (+4298186us) ts_tablet_manager.cc:1151] Deleting tablet 242a77ae57174de48f4eaf43384ee0ba
1214 20:52:26.477995 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.478067 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4298049}
W20251214 20:52:26.478844 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 242a77ae57174de48f4eaf43384ee0ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 242a77ae57174de48f4eaf43384ee0ba already in progress: creating tablet
I20251214 20:52:26.479210 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 239) took 4296 ms. Trace:
I20251214 20:52:26.479318 1856 rpcz_store.cc:276] 1214 20:52:22.182947 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.183006 (+ 59us) service_pool.cc:225] Handling call
1214 20:52:26.479113 (+4296107us) ts_tablet_manager.cc:1151] Deleting tablet 40a9a6192c744282ab5672cbe15fd299
1214 20:52:26.479141 (+ 28us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.479195 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4295922}
W20251214 20:52:26.479768 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 40a9a6192c744282ab5672cbe15fd299 because the tablet was not found. No further retry: Not found: Tablet not found: 40a9a6192c744282ab5672cbe15fd299
I20251214 20:52:26.480147 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 242) took 4182 ms. Trace:
I20251214 20:52:26.480468 1856 rpcz_store.cc:276] 1214 20:52:22.297746 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.297813 (+ 67us) service_pool.cc:225] Handling call
1214 20:52:26.480033 (+4182220us) ts_tablet_manager.cc:1151] Deleting tablet dfe50a6219134431b0a025317932609b
1214 20:52:26.480041 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.480132 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4181915}
W20251214 20:52:26.481058 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet dfe50a6219134431b0a025317932609b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfe50a6219134431b0a025317932609b already in progress: creating tablet
I20251214 20:52:26.481447 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 243) took 4156 ms. Trace:
I20251214 20:52:26.481770 1856 rpcz_store.cc:276] 1214 20:52:22.325095 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.325335 (+ 240us) service_pool.cc:225] Handling call
1214 20:52:26.481343 (+4156008us) ts_tablet_manager.cc:1151] Deleting tablet fe24e4f4e74040249068505058c15622
1214 20:52:26.481352 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.481433 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4155788}
W20251214 20:52:26.482295 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet fe24e4f4e74040249068505058c15622 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24e4f4e74040249068505058c15622 already in progress: creating tablet
I20251214 20:52:26.482637 1856 tablet_replica.cc:333] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:26.483038 1856 raft_consensus.cc:2243] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.483748 1856 raft_consensus.cc:2272] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.484925 1856 ts_tablet_manager.cc:1916] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.490396 1203 tablet_bootstrap.cc:492] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:26.490662 1203 ts_tablet_manager.cc:1403] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.005s
I20251214 20:52:26.491586 1202 tablet_bootstrap.cc:654] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.491592 1203 raft_consensus.cc:359] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.491771 1203 raft_consensus.cc:385] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.491843 1203 raft_consensus.cc:740] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.492081 1203 consensus_queue.cc:260] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.492833 1977 ts_tablet_manager.cc:1929] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.493654 1977 log.cc:1199] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/7dd657f3baff40bb8cb7f4648367f7a1
I20251214 20:52:26.494089 1977 ts_tablet_manager.cc:1950] T 7dd657f3baff40bb8cb7f4648367f7a1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.491796 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet f1eb4c3aa654480caaa69dbd5047365c
I20251214 20:52:26.495175 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 248) took 3300 ms. Trace:
I20251214 20:52:26.495334 1206 tablet_bootstrap.cc:492] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:26.495258 1977 rpcz_store.cc:276] 1214 20:52:23.194501 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.195003 (+ 502us) service_pool.cc:225] Handling call
1214 20:52:26.424315 (+3229312us) ts_tablet_manager.cc:1151] Deleting tablet 7dd657f3baff40bb8cb7f4648367f7a1
1214 20:52:26.424329 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.437873 (+ 13544us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.492715 (+ 54842us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.495156 (+ 2441us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3076928}
I20251214 20:52:26.495661 1206 ts_tablet_manager.cc:1403] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.000s
I20251214 20:52:26.495954 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 7dd657f3baff40bb8cb7f4648367f7a1 (table default.Table_8b86c42f7826440982c88a8625559146 [id=c35009dabd4a42598506e52b3d3e493c]) successfully deleted
I20251214 20:52:26.496501 878 tablet_service.cc:1552] Processing DeleteTablet for tablet f1eb4c3aa654480caaa69dbd5047365c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.496927 1977 tablet_replica.cc:333] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.497185 1977 raft_consensus.cc:2243] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.497324 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet f1eb4c3aa654480caaa69dbd5047365c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.497466 1977 raft_consensus.cc:2272] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.497438 746 tablet_service.cc:1552] Processing DeleteTablet for tablet f1eb4c3aa654480caaa69dbd5047365c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.497361 1206 raft_consensus.cc:359] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.497700 1206 raft_consensus.cc:385] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.497829 1206 raft_consensus.cc:740] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.497947 1203 ts_tablet_manager.cc:1434] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:26.498111 1977 ts_tablet_manager.cc:1916] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.498656 1206 consensus_queue.cc:260] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.501502 1206 ts_tablet_manager.cc:1434] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251214 20:52:26.503755 1203 tablet_bootstrap.cc:492] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:26.504493 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:26.505033 1206 tablet_bootstrap.cc:492] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:26.506439 1982 ts_tablet_manager.cc:1929] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.507373 2437 consensus_queue.cc:1048] T 00cf456eda464f5ba22d6893f73b5cba P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:26.507439 1982 log.cc:1199] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/9d89ee602c524c96a21c0a235af07481
I20251214 20:52:26.508460 1203 tablet_bootstrap.cc:654] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.508746 2327 raft_consensus.cc:493] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:26.508903 2327 raft_consensus.cc:515] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.509581 2327 leader_election.cc:290] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:26.510898 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738668adf2874d6894db6fa875f6990c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:26.510977 1982 ts_tablet_manager.cc:1950] T 9d89ee602c524c96a21c0a235af07481 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:26.511111 1050 raft_consensus.cc:2468] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:26.511854 851 leader_election.cc:304] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:26.512264 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738668adf2874d6894db6fa875f6990c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:26.512804 848 leader_election.cc:343] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.513196 2327 raft_consensus.cc:2804] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:26.513279 2327 raft_consensus.cc:493] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:26.513340 2327 raft_consensus.cc:3060] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.514379 1856 ts_tablet_manager.cc:1929] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.514518 1856 log.cc:1199] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/02e256d4537249a394b80dc36beefef4
I20251214 20:52:26.514832 1856 ts_tablet_manager.cc:1950] T 02e256d4537249a394b80dc36beefef4 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.515179 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 245) took 4123 ms. Trace:
I20251214 20:52:26.515259 1856 rpcz_store.cc:276] 1214 20:52:22.391537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.391680 (+ 143us) service_pool.cc:225] Handling call
1214 20:52:26.482588 (+4090908us) ts_tablet_manager.cc:1151] Deleting tablet 02e256d4537249a394b80dc36beefef4
1214 20:52:26.482596 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.510814 (+ 28218us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.514235 (+ 3421us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.515158 (+ 923us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4090627}
I20251214 20:52:26.515852 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 02e256d4537249a394b80dc36beefef4 (table default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]) successfully deleted
I20251214 20:52:26.515985 2327 raft_consensus.cc:515] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.517108 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738668adf2874d6894db6fa875f6990c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:26.517298 1050 raft_consensus.cc:3060] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.518195 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 263) took 3975 ms. Trace:
I20251214 20:52:26.518277 1856 rpcz_store.cc:276] 1214 20:52:22.542312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.559302 (+ 16990us) service_pool.cc:225] Handling call
1214 20:52:26.517935 (+3958633us) ts_tablet_manager.cc:1151] Deleting tablet 58ac06f852fc4abea5cb213926002ca1
1214 20:52:26.517947 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.518162 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3958027}
I20251214 20:52:26.519590 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 264) took 3958 ms. Trace:
I20251214 20:52:26.519661 1856 rpcz_store.cc:276] 1214 20:52:22.561518 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.563096 (+ 1578us) service_pool.cc:225] Handling call
1214 20:52:26.519470 (+3956374us) ts_tablet_manager.cc:1151] Deleting tablet 6eca34627bbc4a3f9915be72f3128370
1214 20:52:26.519481 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.519573 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3955911}
I20251214 20:52:26.519831 1050 raft_consensus.cc:2468] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
W20251214 20:52:26.520057 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 6eca34627bbc4a3f9915be72f3128370 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6eca34627bbc4a3f9915be72f3128370 already in progress: creating tablet
I20251214 20:52:26.520488 851 leader_election.cc:304] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:26.520866 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 265) took 3527 ms. Trace:
I20251214 20:52:26.520982 2287 raft_consensus.cc:2804] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:26.520995 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 209) took 4191 ms. Trace:
I20251214 20:52:26.521293 2287 raft_consensus.cc:697] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:26.521292 1856 rpcz_store.cc:276] 1214 20:52:22.992978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.993044 (+ 66us) service_pool.cc:225] Handling call
1214 20:52:26.520766 (+3527722us) ts_tablet_manager.cc:1151] Deleting tablet 274185a4115b458e8e9a376bdbcc870b
1214 20:52:26.520775 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.520851 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3520063}
I20251214 20:52:26.521633 2287 consensus_queue.cc:237] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:26.522001 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 274185a4115b458e8e9a376bdbcc870b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 274185a4115b458e8e9a376bdbcc870b already in progress: creating tablet
I20251214 20:52:26.522325 1856 tablet_replica.cc:333] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:26.522807 1856 raft_consensus.cc:2243] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.521659 1982 rpcz_store.cc:276] 1214 20:52:22.329310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.329794 (+ 484us) service_pool.cc:225] Handling call
1214 20:52:26.433333 (+4103539us) ts_tablet_manager.cc:1151] Deleting tablet 9d89ee602c524c96a21c0a235af07481
1214 20:52:26.433346 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.477955 (+ 44609us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.505644 (+ 27689us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.520746 (+ 15102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4103255}
I20251214 20:52:26.523615 1856 raft_consensus.cc:2272] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.523777 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738668adf2874d6894db6fa875f6990c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:26.523908 1206 tablet_bootstrap.cc:654] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.524313 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 9d89ee602c524c96a21c0a235af07481 (table default.Table_66472fe7b4dd4e3eb07839fe78a7bead [id=8872bfac95c64cbba1ccb84d1d9f2483]) successfully deleted
W20251214 20:52:26.524425 848 leader_election.cc:343] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.524792 2327 leader_election.cc:290] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:26.524924 1856 ts_tablet_manager.cc:1916] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.532505 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d9babf534d4b4411aa03906984198232"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:26.533201 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9babf534d4b4411aa03906984198232 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.535482 1856 ts_tablet_manager.cc:1929] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:26.537344 1202 tablet_bootstrap.cc:492] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:26.537676 1202 ts_tablet_manager.cc:1403] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.000s
I20251214 20:52:26.538682 1202 raft_consensus.cc:359] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.538959 1202 raft_consensus.cc:385] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.539068 1202 raft_consensus.cc:740] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.539467 1202 consensus_queue.cc:260] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.540233 1202 ts_tablet_manager.cc:1434] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:26.540611 1202 tablet_bootstrap.cc:492] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:26.543642 1977 ts_tablet_manager.cc:1929] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:26.544528 1977 log.cc:1199] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/023299180d074893847e84dcd9812751
I20251214 20:52:26.544440 1202 tablet_bootstrap.cc:654] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.545529 1856 log.cc:1199] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/5f4760585baf4b4ab3b9b94b5a748a61
I20251214 20:52:26.545821 1977 ts_tablet_manager.cc:1950] T 023299180d074893847e84dcd9812751 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.546216 1856 ts_tablet_manager.cc:1950] T 5f4760585baf4b4ab3b9b94b5a748a61 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.546401 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 251) took 3190 ms. Trace:
I20251214 20:52:26.546550 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 267) took 3546 ms. Trace:
I20251214 20:52:26.546535 1977 rpcz_store.cc:276] 1214 20:52:23.355644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.355691 (+ 47us) service_pool.cc:225] Handling call
1214 20:52:26.496874 (+3141183us) ts_tablet_manager.cc:1151] Deleting tablet 023299180d074893847e84dcd9812751
1214 20:52:26.496885 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.501219 (+ 4334us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.543516 (+ 42297us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.546379 (+ 2863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3140741}
I20251214 20:52:26.546664 1856 rpcz_store.cc:276] 1214 20:52:23.000598 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.000684 (+ 86us) service_pool.cc:225] Handling call
1214 20:52:26.522276 (+3521592us) ts_tablet_manager.cc:1151] Deleting tablet 5f4760585baf4b4ab3b9b94b5a748a61
1214 20:52:26.522285 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.532361 (+ 10076us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.535336 (+ 2975us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.546535 (+ 11199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3521351}
I20251214 20:52:26.547183 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]) successfully deleted
I20251214 20:52:26.547381 1977 tablet_replica.cc:333] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.547619 1977 raft_consensus.cc:2243] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.547240 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]) successfully deleted
I20251214 20:52:26.547869 1977 raft_consensus.cc:2272] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.548460 1977 ts_tablet_manager.cc:1916] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:26.549512 991 consensus_peers.cc:597] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 023299180d074893847e84dcd9812751. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:26.550366 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 266) took 3557 ms. Trace:
I20251214 20:52:26.550892 1856 rpcz_store.cc:276] 1214 20:52:22.993100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.993276 (+ 176us) service_pool.cc:225] Handling call
1214 20:52:26.550178 (+3556902us) ts_tablet_manager.cc:1151] Deleting tablet 3b47ca1a03944cf1847d3578856dafa7
1214 20:52:26.550191 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.550292 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3546482}
W20251214 20:52:26.551642 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3b47ca1a03944cf1847d3578856dafa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b47ca1a03944cf1847d3578856dafa7 already in progress: creating tablet
I20251214 20:52:26.552074 1856 tablet_replica.cc:333] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:26.552553 1856 raft_consensus.cc:2243] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.553081 1856 raft_consensus.cc:2272] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.554492 1856 ts_tablet_manager.cc:1916] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.556030 452 master_service.cc:954] ReplaceTablet: received request to replace tablet 4f358df4d9414dcdb0d197f434e28118 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:26.556471 1977 ts_tablet_manager.cc:1929] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:26.565193 1977 log.cc:1199] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/a38e82f5dd1442f3a1dfcd6fbd94dbaf
I20251214 20:52:26.565676 746 tablet_service.cc:1505] Processing CreateTablet for tablet ed37194008044d2d84421b387f66dbba (DEFAULT_TABLE table=default.Table_691250c9129d453585ad50b32a1e340e [id=edcea358cf1c4046a473b27b94ba56ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.566210 1977 ts_tablet_manager.cc:1950] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.566345 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:26.566579 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:26.566727 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]). Expected version 1 got 0
I20251214 20:52:26.566740 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 253) took 3103 ms. Trace:
I20251214 20:52:26.566823 1977 rpcz_store.cc:276] 1214 20:52:23.462872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.462936 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:26.547251 (+3084315us) ts_tablet_manager.cc:1151] Deleting tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf
1214 20:52:26.547272 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.553113 (+ 5841us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.556378 (+ 3265us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.566721 (+ 10343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3083948}
I20251214 20:52:26.567628 2455 raft_consensus.cc:493] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:26.567847 455 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50088 (request call id 305) took 1182 ms. Trace:
I20251214 20:52:26.567945 2455 raft_consensus.cc:515] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.568181 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 254) took 3076 ms. Trace:
I20251214 20:52:26.568248 1977 rpcz_store.cc:276] 1214 20:52:23.492192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.492283 (+ 91us) service_pool.cc:225] Handling call
1214 20:52:26.568047 (+3075764us) ts_tablet_manager.cc:1151] Deleting tablet d01db63fd5d2474a9defdce0b98fc29d
1214 20:52:26.568058 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.568164 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3062279}
I20251214 20:52:26.567955 455 rpcz_store.cc:276] 1214 20:52:25.385362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.976406 (+591044us) service_pool.cc:225] Handling call
1214 20:52:25.976869 (+ 463us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:25.977120 (+ 251us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:25.977405 (+ 285us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:25.977492 (+ 87us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:26.223708 (+246216us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:26.567781 (+344073us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:26.049257 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:26.049552 (+ 295us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:26.049561 (+ 9us) write_op.cc:435] Acquired schema lock
1214 20:52:26.049564 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:26.049711 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:26.049728 (+ 17us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:26.049733 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:26.049822 (+ 89us) tablet.cc:699] Row locks acquired
1214 20:52:26.049824 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:26.049880 (+ 56us) write_op.cc:270] Start()
1214 20:52:26.049939 (+ 59us) write_op.cc:276] Timestamp: P: 1765745546049785 usec, L: 0
1214 20:52:26.049943 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:26.050246 (+ 303us) log.cc:844] Serialized 405 byte log entry
1214 20:52:26.207942 (+157696us) op_driver.cc:464] REPLICATION: finished
1214 20:52:26.208827 (+ 885us) write_op.cc:301] APPLY: starting
1214 20:52:26.208838 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:26.208870 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:26.208895 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:26.208933 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:26.208957 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:26.208964 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:26.209019 (+ 55us) log.cc:844] Serialized 23 byte log entry
1214 20:52:26.209059 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.209072 (+ 13us) write_op.cc:454] Released schema lock
1214 20:52:26.209076 (+ 4us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:26.209143 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.209147 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":853,"apply.run_cpu_time_us":313,"apply.run_wall_time_us":312,"num_ops":1,"prepare.queue_time_us":71100,"prepare.run_cpu_time_us":1111,"prepare.run_wall_time_us":1615,"raft.queue_time_us":43,"raft.run_cpu_time_us":23,"raft.run_wall_time_us":23,"replication_time_us":157980}]]}
I20251214 20:52:26.569133 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 255) took 3076 ms. Trace:
I20251214 20:52:26.569198 1977 rpcz_store.cc:276] 1214 20:52:23.492347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.492634 (+ 287us) service_pool.cc:225] Handling call
1214 20:52:26.569036 (+3076402us) ts_tablet_manager.cc:1151] Deleting tablet ae948ffc81764692b92045f5befe0184
1214 20:52:26.569044 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.569117 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3063203}
I20251214 20:52:26.569828 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "242a77ae57174de48f4eaf43384ee0ba" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:26.570341 1977 tablet_replica.cc:333] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
W20251214 20:52:26.570377 989 leader_election.cc:343] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.570530 1977 raft_consensus.cc:2243] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.570542 1856 ts_tablet_manager.cc:1929] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.570606 2455 leader_election.cc:290] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:26.570704 1977 raft_consensus.cc:2272] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.571283 1977 ts_tablet_manager.cc:1916] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.570684 1856 log.cc:1199] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/16816fa6b8c34207bb929147910b93cb
I20251214 20:52:26.572114 1856 ts_tablet_manager.cc:1950] T 16816fa6b8c34207bb929147910b93cb P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.572465 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 277) took 3548 ms. Trace:
I20251214 20:52:26.572571 1856 rpcz_store.cc:276] 1214 20:52:23.023839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.023927 (+ 88us) service_pool.cc:225] Handling call
1214 20:52:26.551980 (+3528053us) ts_tablet_manager.cc:1151] Deleting tablet 16816fa6b8c34207bb929147910b93cb
1214 20:52:26.551990 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.562481 (+ 10491us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.570341 (+ 7860us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.572431 (+ 2090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3527850}
I20251214 20:52:26.573019 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 16816fa6b8c34207bb929147910b93cb (table default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]) successfully deleted
I20251214 20:52:26.574501 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 268) took 3551 ms. Trace:
I20251214 20:52:26.575048 1856 rpcz_store.cc:276] 1214 20:52:23.023092 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.024387 (+ 1295us) service_pool.cc:225] Handling call
1214 20:52:26.574325 (+3549938us) ts_tablet_manager.cc:1151] Deleting tablet bc73d590531c409e9f60670435dc8223
1214 20:52:26.574335 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.574455 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3549798}
I20251214 20:52:26.576152 768 raft_consensus.cc:3060] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:26.576367 991 leader_election.cc:343] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 242a77ae57174de48f4eaf43384ee0ba
I20251214 20:52:26.576772 991 leader_election.cc:304] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:26.577382 2167 raft_consensus.cc:2749] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:26.578271 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet bc73d590531c409e9f60670435dc8223 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc73d590531c409e9f60670435dc8223 already in progress: creating tablet
I20251214 20:52:26.578267 1028 tablet_service.cc:1505] Processing CreateTablet for tablet ed37194008044d2d84421b387f66dbba (DEFAULT_TABLE table=default.Table_691250c9129d453585ad50b32a1e340e [id=edcea358cf1c4046a473b27b94ba56ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.578712 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed37194008044d2d84421b387f66dbba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.578684 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 273) took 3555 ms. Trace:
I20251214 20:52:26.578795 1856 rpcz_store.cc:276] 1214 20:52:23.023539 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.025172 (+ 1633us) service_pool.cc:225] Handling call
1214 20:52:26.578552 (+3553380us) ts_tablet_manager.cc:1151] Deleting tablet c8e33ba1149a44c294885d76f676160a
1214 20:52:26.578560 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.578644 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3553240}
W20251214 20:52:26.579268 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c8e33ba1149a44c294885d76f676160a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8e33ba1149a44c294885d76f676160a already in progress: creating tablet
I20251214 20:52:26.580295 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 274) took 3556 ms. Trace:
I20251214 20:52:26.580744 1856 rpcz_store.cc:276] 1214 20:52:23.023614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.025337 (+ 1723us) service_pool.cc:225] Handling call
1214 20:52:26.580167 (+3554830us) ts_tablet_manager.cc:1151] Deleting tablet 1dca369dc5f54a4ea1783e334c5670d6
1214 20:52:26.580177 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.580255 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3554719}
W20251214 20:52:26.581452 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 1dca369dc5f54a4ea1783e334c5670d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1dca369dc5f54a4ea1783e334c5670d6 already in progress: creating tablet
I20251214 20:52:26.581874 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 275) took 3558 ms. Trace:
I20251214 20:52:26.582265 1856 rpcz_store.cc:276] 1214 20:52:23.023680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.025464 (+ 1784us) service_pool.cc:225] Handling call
1214 20:52:26.581771 (+3556307us) ts_tablet_manager.cc:1151] Deleting tablet 762068d560ff47918dd9fd8791083433
1214 20:52:26.581779 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.581831 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3556211}
W20251214 20:52:26.582923 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 762068d560ff47918dd9fd8791083433 because the tablet was not found. No further retry: Not found: Tablet not found: 762068d560ff47918dd9fd8791083433
I20251214 20:52:26.583320 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 276) took 3559 ms. Trace:
I20251214 20:52:26.584492 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf (table default.Table_08488285c9094a9dacc1e489c50271b3 [id=d3b4eef101ec491485c1889f08de9992]) successfully deleted
W20251214 20:52:26.584699 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet d01db63fd5d2474a9defdce0b98fc29d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d01db63fd5d2474a9defdce0b98fc29d already in progress: creating tablet
W20251214 20:52:26.584877 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ae948ffc81764692b92045f5befe0184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae948ffc81764692b92045f5befe0184 already in progress: creating tablet
I20251214 20:52:26.590566 1206 tablet_bootstrap.cc:492] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:26.590742 1206 ts_tablet_manager.cc:1403] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.087s user 0.008s sys 0.001s
I20251214 20:52:26.591557 1206 raft_consensus.cc:359] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.591724 1206 raft_consensus.cc:385] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.591773 1206 raft_consensus.cc:740] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.591881 749 tablet_service.cc:1505] Processing CreateTablet for tablet 59ccb6c3b35044dc913cce74b5802e38 (DEFAULT_TABLE table=default.Table_7e2ed71ac05340a1be0ebd6af1b9dcc4 [id=b5622661ea9f40f7ac11f44ae92a5194]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.591941 1206 consensus_queue.cc:260] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.592163 878 tablet_service.cc:1505] Processing CreateTablet for tablet ed37194008044d2d84421b387f66dbba (DEFAULT_TABLE table=default.Table_691250c9129d453585ad50b32a1e340e [id=edcea358cf1c4046a473b27b94ba56ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.592406 1206 ts_tablet_manager.cc:1434] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:26.592726 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed37194008044d2d84421b387f66dbba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.597043 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 59ccb6c3b35044dc913cce74b5802e38 (DEFAULT_TABLE table=default.Table_7e2ed71ac05340a1be0ebd6af1b9dcc4 [id=b5622661ea9f40f7ac11f44ae92a5194]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.597568 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ccb6c3b35044dc913cce74b5802e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.599552 879 tablet_service.cc:1505] Processing CreateTablet for tablet 59ccb6c3b35044dc913cce74b5802e38 (DEFAULT_TABLE table=default.Table_7e2ed71ac05340a1be0ebd6af1b9dcc4 [id=b5622661ea9f40f7ac11f44ae92a5194]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.599566 1202 tablet_bootstrap.cc:492] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:26.600050 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ccb6c3b35044dc913cce74b5802e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.600539 1202 ts_tablet_manager.cc:1403] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.005s
I20251214 20:52:26.610046 747 tablet_service.cc:1505] Processing CreateTablet for tablet ed9790fbebda4f848291930bf1dd8174 (DEFAULT_TABLE table=default.Table_0945dd902049447298aef35db32f33ba [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.607898 1202 raft_consensus.cc:359] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.611238 1202 raft_consensus.cc:385] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.611748 1202 raft_consensus.cc:740] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.612507 884 tablet_service.cc:1505] Processing CreateTablet for tablet ed9790fbebda4f848291930bf1dd8174 (DEFAULT_TABLE table=default.Table_0945dd902049447298aef35db32f33ba [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.612869 1202 consensus_queue.cc:260] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.612876 1977 ts_tablet_manager.cc:1929] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:26.616451 1977 log.cc:1199] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/2b92b6d5dde2465684d19aee44a83160
I20251214 20:52:26.616783 1977 ts_tablet_manager.cc:1950] T 2b92b6d5dde2465684d19aee44a83160 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.617089 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 257) took 3066 ms. Trace:
I20251214 20:52:26.617166 1977 rpcz_store.cc:276] 1214 20:52:23.551155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.551207 (+ 52us) service_pool.cc:225] Handling call
1214 20:52:26.570298 (+3019091us) ts_tablet_manager.cc:1151] Deleting tablet 2b92b6d5dde2465684d19aee44a83160
1214 20:52:26.570307 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.609929 (+ 39622us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.612797 (+ 2868us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.617071 (+ 4274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3018884}
I20251214 20:52:26.617594 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 261) took 2997 ms. Trace:
I20251214 20:52:26.612988 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed9790fbebda4f848291930bf1dd8174. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.618175 1202 ts_tablet_manager.cc:1434] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20251214 20:52:26.619624 2478 tablet.cc:1722] T 88b9ca42a5ce4572aebe2a24f4758b12 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:26.620306 1202 tablet_bootstrap.cc:492] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:26.617668 1977 rpcz_store.cc:276] 1214 20:52:23.619675 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.622934 (+ 3259us) service_pool.cc:225] Handling call
1214 20:52:26.617452 (+2994518us) ts_tablet_manager.cc:1151] Deleting tablet 37a33cc9bb784f72956ac9f5493a0c3d
1214 20:52:26.617461 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.617579 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2994236}
I20251214 20:52:26.626461 1202 tablet_bootstrap.cc:654] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.627888 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 2b92b6d5dde2465684d19aee44a83160 (table default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]) successfully deleted
W20251214 20:52:26.628324 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 37a33cc9bb784f72956ac9f5493a0c3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37a33cc9bb784f72956ac9f5493a0c3d already in progress: creating tablet
I20251214 20:52:26.636888 2499 tablet.cc:1722] T 88b9ca42a5ce4572aebe2a24f4758b12 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:26.640774 748 tablet_service.cc:1505] Processing CreateTablet for tablet 894e3f27b7904db994711aa80a9b1600 (DEFAULT_TABLE table=default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.641285 880 tablet_service.cc:1505] Processing CreateTablet for tablet 894e3f27b7904db994711aa80a9b1600 (DEFAULT_TABLE table=default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.641343 2364 raft_consensus.cc:493] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:26.641194 1018 tablet_service.cc:1505] Processing CreateTablet for tablet ed9790fbebda4f848291930bf1dd8174 (DEFAULT_TABLE table=default.Table_0945dd902049447298aef35db32f33ba [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.641494 2364 raft_consensus.cc:515] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.641743 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed9790fbebda4f848291930bf1dd8174. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.642151 2364 leader_election.cc:290] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:26.642944 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "190afbb866854dd7b5393e04443752b7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:26.643626 989 leader_election.cc:343] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.648207 743 tablet_service.cc:1505] Processing CreateTablet for tablet 47e873c6be06401282f60d87d1a70865 (DEFAULT_TABLE table=default.Table_b58a6dfd93fb438684f7564325241f3e [id=e85a8088178744e0a19699f5d680b4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.648499 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 47e873c6be06401282f60d87d1a70865 (DEFAULT_TABLE table=default.Table_b58a6dfd93fb438684f7564325241f3e [id=e85a8088178744e0a19699f5d680b4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.648977 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e873c6be06401282f60d87d1a70865. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.649488 2498 tablet.cc:1722] T 88b9ca42a5ce4572aebe2a24f4758b12 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:26.655584 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 894e3f27b7904db994711aa80a9b1600 (DEFAULT_TABLE table=default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.655586 883 tablet_service.cc:1505] Processing CreateTablet for tablet 47e873c6be06401282f60d87d1a70865 (DEFAULT_TABLE table=default.Table_b58a6dfd93fb438684f7564325241f3e [id=e85a8088178744e0a19699f5d680b4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.656287 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894e3f27b7904db994711aa80a9b1600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.658762 1856 rpcz_store.cc:276] 1214 20:52:23.023758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.025579 (+ 1821us) service_pool.cc:225] Handling call
1214 20:52:26.583222 (+3557643us) ts_tablet_manager.cc:1151] Deleting tablet 0cafed2861e94cd9b0d10d33864ba45d
1214 20:52:26.583230 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.583281 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3557549}
W20251214 20:52:26.659484 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 0cafed2861e94cd9b0d10d33864ba45d because the tablet was not found. No further retry: Not found: Tablet not found: 0cafed2861e94cd9b0d10d33864ba45d
I20251214 20:52:26.665478 1202 tablet_bootstrap.cc:492] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:26.665848 1202 ts_tablet_manager.cc:1403] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.019s
I20251214 20:52:26.666955 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 268) took 2995 ms. Trace:
I20251214 20:52:26.667080 1977 rpcz_store.cc:276] 1214 20:52:23.671868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.673814 (+ 1946us) service_pool.cc:225] Handling call
1214 20:52:26.624201 (+2950387us) ts_tablet_manager.cc:1151] Deleting tablet fc7353593bda4fceaa389c393ddc19c6
1214 20:52:26.665058 (+ 40857us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.666921 (+ 1863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2950142}
W20251214 20:52:26.667615 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet fc7353593bda4fceaa389c393ddc19c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc7353593bda4fceaa389c393ddc19c6 already in progress: creating tablet
I20251214 20:52:26.667876 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "190afbb866854dd7b5393e04443752b7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:26.668118 904 raft_consensus.cc:2468] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:26.668651 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 278) took 3644 ms. Trace:
I20251214 20:52:26.668740 1856 rpcz_store.cc:276] 1214 20:52:23.023920 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.025689 (+ 1769us) service_pool.cc:225] Handling call
1214 20:52:26.668515 (+3642826us) ts_tablet_manager.cc:1151] Deleting tablet 0d7c7b3f62764ff497f4d70b53eed7de
1214 20:52:26.668526 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.668628 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3642721}
W20251214 20:52:26.669198 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 0d7c7b3f62764ff497f4d70b53eed7de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d7c7b3f62764ff497f4d70b53eed7de already in progress: creating tablet
I20251214 20:52:26.671377 991 leader_election.cc:304] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:26.672053 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894e3f27b7904db994711aa80a9b1600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.672534 2439 log.cc:826] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Log is configured to *not* fsync() on all Append() calls
I20251214 20:52:26.674083 2167 raft_consensus.cc:2804] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:26.674192 2167 raft_consensus.cc:493] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:26.674254 2167 raft_consensus.cc:3060] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.675271 1982 tablet_replica.cc:333] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:26.675514 1982 raft_consensus.cc:2243] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.675550 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed37194008044d2d84421b387f66dbba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.675696 1982 raft_consensus.cc:2272] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.676111 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59ccb6c3b35044dc913cce74b5802e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.679666 1206 tablet_bootstrap.cc:492] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:26.681552 1206 tablet_bootstrap.cc:654] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.682662 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed9790fbebda4f848291930bf1dd8174. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.684518 1982 ts_tablet_manager.cc:1916] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.684588 2167 raft_consensus.cc:515] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.685880 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "190afbb866854dd7b5393e04443752b7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:26.685943 2167 leader_election.cc:290] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:26.686453 989 leader_election.cc:343] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.687314 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e873c6be06401282f60d87d1a70865. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.687721 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 269) took 3664 ms. Trace:
I20251214 20:52:26.687860 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894e3f27b7904db994711aa80a9b1600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.687806 1856 rpcz_store.cc:276] 1214 20:52:23.023224 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.026596 (+ 3372us) service_pool.cc:225] Handling call
1214 20:52:26.687575 (+3660979us) ts_tablet_manager.cc:1151] Deleting tablet f342dafae1d04be3ba38f7c0e4f5d058
1214 20:52:26.687588 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.687700 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3660806}
W20251214 20:52:26.688364 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f342dafae1d04be3ba38f7c0e4f5d058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f342dafae1d04be3ba38f7c0e4f5d058 already in progress: creating tablet
I20251214 20:52:26.689613 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 282) took 3665 ms. Trace:
I20251214 20:52:26.689685 1856 rpcz_store.cc:276] 1214 20:52:23.024237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.027177 (+ 2940us) service_pool.cc:225] Handling call
1214 20:52:26.689496 (+3662319us) ts_tablet_manager.cc:1151] Deleting tablet eb29fd8c85344ed8aa8d880e2ed240bc
1214 20:52:26.689508 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.689595 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3662176}
W20251214 20:52:26.690086 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet eb29fd8c85344ed8aa8d880e2ed240bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb29fd8c85344ed8aa8d880e2ed240bc already in progress: creating tablet
I20251214 20:52:26.691190 746 tablet_service.cc:1505] Processing CreateTablet for tablet c86bb092060b4019ac0f557bfd3f14f5 (DEFAULT_TABLE table=default.Table_09f08f5ffa814302a74b268170036a24 [id=71cefadd096649a8aca97ece15ff1bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.691612 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86bb092060b4019ac0f557bfd3f14f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.691411 887 tablet_service.cc:1505] Processing CreateTablet for tablet c86bb092060b4019ac0f557bfd3f14f5 (DEFAULT_TABLE table=default.Table_09f08f5ffa814302a74b268170036a24 [id=71cefadd096649a8aca97ece15ff1bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.692478 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86bb092060b4019ac0f557bfd3f14f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.700662 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_79c0f2d73838484eba9470ccec3e90b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:26.701459 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79c0f2d73838484eba9470ccec3e90b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.703919 1030 tablet_service.cc:1505] Processing CreateTablet for tablet c86bb092060b4019ac0f557bfd3f14f5 (DEFAULT_TABLE table=default.Table_09f08f5ffa814302a74b268170036a24 [id=71cefadd096649a8aca97ece15ff1bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.704416 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86bb092060b4019ac0f557bfd3f14f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.715853 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 286) took 3630 ms. Trace:
I20251214 20:52:26.716624 1856 rpcz_store.cc:276] 1214 20:52:23.085412 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.090824 (+ 5412us) service_pool.cc:225] Handling call
1214 20:52:26.715722 (+3624898us) ts_tablet_manager.cc:1151] Deleting tablet 604b7c6fe3b24daca70711a1990af556
1214 20:52:26.715735 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.715824 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3624648}
W20251214 20:52:26.717486 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 604b7c6fe3b24daca70711a1990af556 because the tablet was not found. No further retry: Not found: Tablet not found: 604b7c6fe3b24daca70711a1990af556
I20251214 20:52:26.717900 1856 tablet_replica.cc:333] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:26.718324 1856 raft_consensus.cc:2243] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.718880 2468 raft_consensus.cc:493] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:26.719041 2468 raft_consensus.cc:515] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.719424 1856 raft_consensus.cc:2272] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.719653 2468 leader_election.cc:290] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:26.720371 1856 ts_tablet_manager.cc:1916] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:26.720510 713 leader_election.cc:343] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: ca263a80702f4f64aa7c872c2be08207
W20251214 20:52:26.720674 712 leader_election.cc:343] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: ca263a80702f4f64aa7c872c2be08207
I20251214 20:52:26.720832 713 leader_election.cc:304] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:26.721241 2468 raft_consensus.cc:2749] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:26.726923 1023 tablet_service.cc:1505] Processing CreateTablet for tablet b2bea617954c475bac7e9b8719faa028 (DEFAULT_TABLE table=default.Table_6a7108611108447db74a6feddaeea2b5 [id=0a848dd5f6474712b2eba70add3743dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.727443 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2bea617954c475bac7e9b8719faa028. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.732682 1206 tablet_bootstrap.cc:492] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:26.732870 1206 ts_tablet_manager.cc:1403] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20251214 20:52:26.733006 1982 ts_tablet_manager.cc:1929] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:26.733355 1982 log.cc:1199] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/4d6e7306254044449f1127fb963ca5f0
I20251214 20:52:26.733727 1982 ts_tablet_manager.cc:1950] T 4d6e7306254044449f1127fb963ca5f0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:26.733767 1206 raft_consensus.cc:359] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.733968 1206 raft_consensus.cc:385] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.734038 1206 raft_consensus.cc:740] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.734246 1206 consensus_queue.cc:260] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.734877 1206 ts_tablet_manager.cc:1434] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:26.735392 1206 tablet_bootstrap.cc:492] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:26.737277 1206 tablet_bootstrap.cc:654] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.737624 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_bbd8d7f1a4b04eeeaf8ccd7e060b7476"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:26.738425 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbd8d7f1a4b04eeeaf8ccd7e060b7476 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.738667 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 205) took 4409 ms. Trace:
I20251214 20:52:26.667822 1202 raft_consensus.cc:359] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.739624 1202 raft_consensus.cc:385] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.738942 1982 rpcz_store.cc:276] 1214 20:52:22.328907 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.329937 (+ 1030us) service_pool.cc:225] Handling call
1214 20:52:26.524580 (+4194643us) ts_tablet_manager.cc:1151] Deleting tablet 4d6e7306254044449f1127fb963ca5f0
1214 20:52:26.675181 (+150601us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.699030 (+ 23849us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.732370 (+ 33340us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.738558 (+ 6188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4194287}
I20251214 20:52:26.740124 1202 raft_consensus.cc:740] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.741281 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 212) took 4409 ms. Trace:
I20251214 20:52:26.742238 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 4d6e7306254044449f1127fb963ca5f0 (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]) successfully deleted
I20251214 20:52:26.741582 1202 consensus_queue.cc:260] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.741694 1982 rpcz_store.cc:276] 1214 20:52:22.331537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.331583 (+ 46us) service_pool.cc:225] Handling call
1214 20:52:26.740630 (+4409047us) ts_tablet_manager.cc:1151] Deleting tablet b1df0dad2fb040a5a12e0401e00c87db
1214 20:52:26.740642 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.741041 (+ 399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4408725}
I20251214 20:52:26.743433 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 213) took 4411 ms. Trace:
I20251214 20:52:26.743763 1982 rpcz_store.cc:276] 1214 20:52:22.331707 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.331771 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:26.743073 (+4411302us) ts_tablet_manager.cc:1151] Deleting tablet 2b48bb260e854beb9ec61ce9c79b46c2
1214 20:52:26.743083 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.743416 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4411016}
I20251214 20:52:26.744303 1202 ts_tablet_manager.cc:1434] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.078s user 0.002s sys 0.001s
I20251214 20:52:26.744629 747 tablet_service.cc:1505] Processing CreateTablet for tablet b2bea617954c475bac7e9b8719faa028 (DEFAULT_TABLE table=default.Table_6a7108611108447db74a6feddaeea2b5 [id=0a848dd5f6474712b2eba70add3743dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.745002 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2bea617954c475bac7e9b8719faa028. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.744906 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 214) took 4413 ms. Trace:
I20251214 20:52:26.745252 1202 tablet_bootstrap.cc:492] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:26.745152 1982 rpcz_store.cc:276] 1214 20:52:22.331816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.331921 (+ 105us) service_pool.cc:225] Handling call
1214 20:52:26.744517 (+4412596us) ts_tablet_manager.cc:1151] Deleting tablet 012b81223d0f4ae9b742014e329f11a9
1214 20:52:26.744524 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.744794 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4412472}
W20251214 20:52:26.745728 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b1df0dad2fb040a5a12e0401e00c87db because the tablet was not found. No further retry: Not found: Tablet not found: b1df0dad2fb040a5a12e0401e00c87db
I20251214 20:52:26.745834 1206 tablet_bootstrap.cc:492] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:26.745965 1206 ts_tablet_manager.cc:1403] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
W20251214 20:52:26.745967 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2b48bb260e854beb9ec61ce9c79b46c2 because the tablet was not found. No further retry: Not found: Tablet not found: 2b48bb260e854beb9ec61ce9c79b46c2
W20251214 20:52:26.746201 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 012b81223d0f4ae9b742014e329f11a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012b81223d0f4ae9b742014e329f11a9 already in progress: creating tablet
I20251214 20:52:26.746222 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 215) took 4414 ms. Trace:
I20251214 20:52:26.746485 1982 rpcz_store.cc:276] 1214 20:52:22.331944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.332059 (+ 115us) service_pool.cc:225] Handling call
1214 20:52:26.745941 (+4413882us) ts_tablet_manager.cc:1151] Deleting tablet 98b7cac3930a4a9eb16ac413c8f1e092
1214 20:52:26.745950 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.746204 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4413675}
W20251214 20:52:26.747274 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 98b7cac3930a4a9eb16ac413c8f1e092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98b7cac3930a4a9eb16ac413c8f1e092 already in progress: creating tablet
I20251214 20:52:26.747450 1982 tablet_replica.cc:333] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:26.748107 1982 raft_consensus.cc:2243] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.748807 1982 raft_consensus.cc:2272] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.749542 452 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 4f358df4d9414dcdb0d197f434e28118
I20251214 20:52:26.749979 452 catalog_manager.cc:6724] ReplaceTablet: tablet 4f358df4d9414dcdb0d197f434e28118 deleted and replaced by tablet a4bcac304fd14e20b0607c23d1569a31
I20251214 20:52:26.750573 1982 ts_tablet_manager.cc:1916] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.750777 1202 tablet_bootstrap.cc:654] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.751778 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f358df4d9414dcdb0d197f434e28118 with delete_type TABLET_DATA_DELETED (replaced by tablet a4bcac304fd14e20b0607c23d1569a31) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.752005 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f358df4d9414dcdb0d197f434e28118 with delete_type TABLET_DATA_DELETED (replaced by tablet a4bcac304fd14e20b0607c23d1569a31) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.753719 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e873c6be06401282f60d87d1a70865. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.758307 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f358df4d9414dcdb0d197f434e28118 with delete_type TABLET_DATA_DELETED (replaced by tablet a4bcac304fd14e20b0607c23d1569a31) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.758677 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 272) took 3008 ms. Trace:
I20251214 20:52:26.758751 1977 rpcz_store.cc:276] 1214 20:52:23.750670 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.750721 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:26.668395 (+2917674us) ts_tablet_manager.cc:1151] Deleting tablet 880a78b924574181bdd804d599f4f579
1214 20:52:26.758514 (+ 90119us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.758658 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2917355}
I20251214 20:52:26.746798 1206 raft_consensus.cc:359] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.759163 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 275) took 2989 ms. Trace:
I20251214 20:52:26.759243 1206 raft_consensus.cc:385] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.759231 1977 rpcz_store.cc:276] 1214 20:52:23.769639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.769695 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:26.759032 (+2989337us) ts_tablet_manager.cc:1151] Deleting tablet e68bf6b1a2974957adae70449334df64
1214 20:52:26.759045 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.759147 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2989132}
I20251214 20:52:26.759335 1206 raft_consensus.cc:740] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.759552 1206 consensus_queue.cc:260] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.759896 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 279) took 2917 ms. Trace:
I20251214 20:52:26.759974 1977 rpcz_store.cc:276] 1214 20:52:23.842071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.842124 (+ 53us) service_pool.cc:225] Handling call
1214 20:52:26.759806 (+2917682us) ts_tablet_manager.cc:1151] Deleting tablet 825baba561ee4a8ebd44f75d8c51706a
1214 20:52:26.759814 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.759882 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2917440}
I20251214 20:52:26.760637 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 282) took 2848 ms. Trace:
I20251214 20:52:26.760717 1977 rpcz_store.cc:276] 1214 20:52:23.911802 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.911892 (+ 90us) service_pool.cc:225] Handling call
1214 20:52:26.760545 (+2848653us) ts_tablet_manager.cc:1151] Deleting tablet b69bb4d5979f492683fc13238a668b9f
1214 20:52:26.760553 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.760622 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2848365}
I20251214 20:52:26.761046 1977 tablet_replica.cc:333] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:26.761229 1206 ts_tablet_manager.cc:1434] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251214 20:52:26.761229 1977 raft_consensus.cc:2243] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.761397 1977 raft_consensus.cc:2272] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.761749 2287 raft_consensus.cc:493] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:26.761893 2287 raft_consensus.cc:515] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.761976 1977 ts_tablet_manager.cc:1916] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.762521 2287 leader_election.cc:290] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:26.763145 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba7952290e2c4746802cffeb14971da0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:26.763638 848 leader_election.cc:343] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.764719 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba7952290e2c4746802cffeb14971da0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:26.764909 1050 raft_consensus.cc:2468] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:26.765292 851 leader_election.cc:304] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:26.765548 2287 raft_consensus.cc:2804] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:26.765612 2287 raft_consensus.cc:493] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:26.765667 2287 raft_consensus.cc:3060] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:26.766073 989 consensus_peers.cc:597] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:26.767457 1206 tablet_bootstrap.cc:492] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:26.768054 2287 raft_consensus.cc:515] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.768203 452 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_0dbf8224a9254c70a34ecf1cbcc79306" } new_table_name: "default.Table_ccc4e873bf4247069a3b9bc25d2a0ca6" modify_external_catalogs: true
W20251214 20:52:26.768735 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 880a78b924574181bdd804d599f4f579 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 880a78b924574181bdd804d599f4f579 already in progress: creating tablet
I20251214 20:52:26.768849 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba7952290e2c4746802cffeb14971da0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:26.769281 848 leader_election.cc:343] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:26.769037 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet e68bf6b1a2974957adae70449334df64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e68bf6b1a2974957adae70449334df64 already in progress: creating tablet
W20251214 20:52:26.770099 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 825baba561ee4a8ebd44f75d8c51706a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 825baba561ee4a8ebd44f75d8c51706a already in progress: creating tablet
W20251214 20:52:26.770375 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet b69bb4d5979f492683fc13238a668b9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b69bb4d5979f492683fc13238a668b9f already in progress: creating tablet
I20251214 20:52:26.770460 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba7952290e2c4746802cffeb14971da0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:26.770612 1050 raft_consensus.cc:3060] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.770993 884 tablet_service.cc:1505] Processing CreateTablet for tablet 641a9073c73d41c09e07cabc60735ba4 (DEFAULT_TABLE table=default.Table_fd97b3b527e240fbb6d62d3c33059904 [id=c1ab5c38b5ee4129a8bccf874894ddcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.773506 1050 raft_consensus.cc:2468] T ba7952290e2c4746802cffeb14971da0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:26.773713 2287 leader_election.cc:290] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:26.774339 851 leader_election.cc:304] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:26.774803 2287 raft_consensus.cc:2804] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:26.775096 2287 raft_consensus.cc:697] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:26.775347 1856 ts_tablet_manager.cc:1929] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:26.775489 1856 log.cc:1199] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/023299180d074893847e84dcd9812751
I20251214 20:52:26.775771 2287 consensus_queue.cc:237] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:26.777014 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 641a9073c73d41c09e07cabc60735ba4 (DEFAULT_TABLE table=default.Table_fd97b3b527e240fbb6d62d3c33059904 [id=c1ab5c38b5ee4129a8bccf874894ddcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.778107 1856 ts_tablet_manager.cc:1950] T 023299180d074893847e84dcd9812751 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.778470 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 296) took 3442 ms. Trace:
I20251214 20:52:26.778568 1856 rpcz_store.cc:276] 1214 20:52:23.336401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.374344 (+ 37943us) service_pool.cc:225] Handling call
1214 20:52:26.717846 (+3343502us) ts_tablet_manager.cc:1151] Deleting tablet 023299180d074893847e84dcd9812751
1214 20:52:26.717855 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.748000 (+ 30145us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.751115 (+ 3115us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.778453 (+ 27338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3342956}
I20251214 20:52:26.778908 1856 tablet_replica.cc:333] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:26.779099 1856 raft_consensus.cc:2243] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.779117 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]) successfully deleted
I20251214 20:52:26.779369 1856 pending_rounds.cc:70] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:26.779414 1856 pending_rounds.cc:77] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493757677428736 op_type: NO_OP noop_request { }
I20251214 20:52:26.779508 1856 raft_consensus.cc:2889] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:26.779580 1856 raft_consensus.cc:2272] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.779753 1206 tablet_bootstrap.cc:654] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.779933 1203 tablet_bootstrap.cc:492] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:26.780056 1203 ts_tablet_manager.cc:1403] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.276s user 0.024s sys 0.031s
I20251214 20:52:26.780880 1203 raft_consensus.cc:359] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.781021 1203 raft_consensus.cc:385] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.781068 1203 raft_consensus.cc:740] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.781292 1203 consensus_queue.cc:260] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.781913 1203 ts_tablet_manager.cc:1434] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:26.782426 1203 tablet_bootstrap.cc:492] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:26.784665 883 tablet_service.cc:1505] Processing CreateTablet for tablet b2bea617954c475bac7e9b8719faa028 (DEFAULT_TABLE table=default.Table_6a7108611108447db74a6feddaeea2b5 [id=0a848dd5f6474712b2eba70add3743dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.787096 748 tablet_service.cc:1505] Processing CreateTablet for tablet 641a9073c73d41c09e07cabc60735ba4 (DEFAULT_TABLE table=default.Table_fd97b3b527e240fbb6d62d3c33059904 [id=c1ab5c38b5ee4129a8bccf874894ddcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:26.787509 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641a9073c73d41c09e07cabc60735ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.778673 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641a9073c73d41c09e07cabc60735ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.790545 1203 tablet_bootstrap.cc:654] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.795336 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "190afbb866854dd7b5393e04443752b7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:26.795526 903 raft_consensus.cc:3060] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.797976 903 raft_consensus.cc:2468] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:26.798885 991 leader_election.cc:304] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:26.799248 2167 raft_consensus.cc:2804] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:26.799453 2167 raft_consensus.cc:697] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:26.799768 2167 consensus_queue.cc:237] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.798233 1856 ts_tablet_manager.cc:1916] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.802938 1203 tablet_bootstrap.cc:492] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:26.803090 1203 ts_tablet_manager.cc:1403] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251214 20:52:26.803907 1203 raft_consensus.cc:359] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.804081 1203 raft_consensus.cc:385] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.804131 1203 raft_consensus.cc:740] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.804309 1203 consensus_queue.cc:260] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.804881 1203 ts_tablet_manager.cc:1434] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:26.805156 1203 tablet_bootstrap.cc:492] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:26.806728 1203 tablet_bootstrap.cc:654] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.812423 450 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:26.813609 454 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_3a068835084e406eb4047a827faeb5d7 [id=bdb017130eed4d76b635603560c91d41] alter complete (version 2)
W20251214 20:52:26.814877 454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:39607 (request call id 164) took 1390 ms (client timeout 1000 ms). Trace:
W20251214 20:52:26.814980 454 rpcz_store.cc:269] 1214 20:52:25.424341 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.474714 (+ 50373us) service_pool.cc:225] Handling call
1214 20:52:25.483169 (+ 8455us) spinlock_profiling.cc:243] Waited 7.51 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b17aa08 00007ff411c87ca6 00007ff411c879d5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:26.814807 (+1331638us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:25.916236 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:25.916308 (+ 72us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:25.916312 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:25.916314 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:25.916397 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:25.916407 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:25.916411 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:25.916458 (+ 47us) tablet.cc:699] Row locks acquired
1214 20:52:25.916460 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:25.916487 (+ 27us) write_op.cc:270] Start()
1214 20:52:25.916512 (+ 25us) write_op.cc:276] Timestamp: P: 1765745545916426 usec, L: 0
1214 20:52:25.916515 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:25.916650 (+ 135us) log.cc:844] Serialized 356 byte log entry
1214 20:52:26.085871 (+169221us) op_driver.cc:464] REPLICATION: finished
1214 20:52:26.086901 (+ 1030us) write_op.cc:301] APPLY: starting
1214 20:52:26.086912 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:26.086939 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:26.086959 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:26.087002 (+ 43us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:26.087027 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:26.087034 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:26.087086 (+ 52us) log.cc:844] Serialized 23 byte log entry
1214 20:52:26.087155 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.087168 (+ 13us) write_op.cc:454] Released schema lock
1214 20:52:26.087172 (+ 4us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:26.087237 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.087241 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1214 20:52:26.114659 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:26.114813 (+ 154us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:26.114822 (+ 9us) write_op.cc:435] Acquired schema lock
1214 20:52:26.114825 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:26.114963 (+ 138us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:26.114979 (+ 16us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:26.114984 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:26.115068 (+ 84us) tablet.cc:699] Row locks acquired
1214 20:52:26.115070 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:26.115155 (+ 85us) write_op.cc:270] Start()
1214 20:52:26.115205 (+ 50us) write_op.cc:276] Timestamp: P: 1765745546115018 usec, L: 0
1214 20:52:26.115208 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:26.180473 (+ 65265us) log.cc:844] Serialized 356 byte log entry
1214 20:52:26.181159 (+ 686us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 000000000098492a 00007ff413306922 00007ff4146e9875 00007ff4146e8399 00007ff4146f07c0 00007ff4146f05cd 0000000000a8970c 00007ff40748e20f 00007ff40749152b 00007ff4074913a8 0000000000a8970c
1214 20:52:26.281194 (+100035us) op_driver.cc:464] REPLICATION: finished
1214 20:52:26.282232 (+ 1038us) write_op.cc:301] APPLY: starting
1214 20:52:26.282261 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:26.282321 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:26.282324 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:26.282408 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:26.282454 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:26.282462 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:26.282583 (+ 121us) log.cc:844] Serialized 23 byte log entry
1214 20:52:26.282686 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.282712 (+ 26us) write_op.cc:454] Released schema lock
1214 20:52:26.282721 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:26.282824 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.282828 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":17274752,"child_traces":[["op",{"apply.queue_time_us":997,"apply.run_cpu_time_us":328,"apply.run_wall_time_us":329,"num_ops":1,"prepare.queue_time_us":20075,"prepare.run_cpu_time_us":490,"prepare.run_wall_time_us":597,"raft.queue_time_us":93,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":2,"replication_time_us":169339}],["op",{"apply.queue_time_us":930,"apply.run_cpu_time_us":595,"apply.run_wall_time_us":596,"num_ops":1,"prepare.queue_time_us":25719,"prepare.run_cpu_time_us":1662,"prepare.run_wall_time_us":66671,"raft.queue_time_us":48,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":165947,"spinlock_wait_cycles":149508352}]]}
I20251214 20:52:26.816258 454 master_service.cc:954] ReplaceTablet: received request to replace tablet 47e813373fbd482298312ead4f037c01 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:26.816540 446 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_89c0478b97144f35b1a0adeb64be2d9f [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde] alter complete (version 1)
I20251214 20:52:26.817097 1203 tablet_bootstrap.cc:492] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:26.817540 1203 ts_tablet_manager.cc:1403] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20251214 20:52:26.818527 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet ca57963bcfb54eec9fcbed82757d8387
I20251214 20:52:26.819001 453 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 302) took 1613 ms. Trace:
I20251214 20:52:26.818904 1203 raft_consensus.cc:359] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.819252 1203 raft_consensus.cc:385] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.819342 1203 raft_consensus.cc:740] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.819118 453 rpcz_store.cc:276] 1214 20:52:25.205368 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.927835 (+722467us) service_pool.cc:225] Handling call
1214 20:52:25.928361 (+ 526us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:25.928506 (+ 145us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:25.928581 (+ 75us) catalog_manager.cc:2824] Locking tablets
1214 20:52:26.221930 (+293349us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:26.305443 (+ 83513us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:26.305544 (+ 101us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:26.818457 (+512913us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:26.818964 (+ 507us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:26.245316 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:26.245515 (+ 199us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:26.245526 (+ 11us) write_op.cc:435] Acquired schema lock
1214 20:52:26.245529 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:26.245700 (+ 171us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:26.245718 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:26.245724 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:26.245869 (+ 145us) tablet.cc:699] Row locks acquired
1214 20:52:26.245872 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:26.245930 (+ 58us) write_op.cc:270] Start()
1214 20:52:26.245983 (+ 53us) write_op.cc:276] Timestamp: P: 1765745546245897 usec, L: 0
1214 20:52:26.245987 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:26.246309 (+ 322us) log.cc:844] Serialized 790 byte log entry
1214 20:52:26.284811 (+ 38502us) op_driver.cc:464] REPLICATION: finished
1214 20:52:26.297293 (+ 12482us) write_op.cc:301] APPLY: starting
1214 20:52:26.297322 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:26.297370 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:26.304747 (+ 7377us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:26.304855 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:26.304900 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:26.304907 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:26.305012 (+ 105us) log.cc:844] Serialized 29 byte log entry
1214 20:52:26.305088 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.305116 (+ 28us) write_op.cc:454] Released schema lock
1214 20:52:26.305125 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:26.305219 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.305223 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":12423,"apply.run_cpu_time_us":610,"apply.run_wall_time_us":7918,"num_ops":2,"prepare.queue_time_us":22502,"prepare.run_cpu_time_us":1132,"prepare.run_wall_time_us":1575,"replication_time_us":38796}]]}
I20251214 20:52:26.819718 1203 consensus_queue.cc:260] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.820865 1203 ts_tablet_manager.cc:1434] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:26.821327 1203 tablet_bootstrap.cc:492] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:26.821640 1202 tablet_bootstrap.cc:492] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:26.821758 1202 ts_tablet_manager.cc:1403] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.000s
I20251214 20:52:26.822441 1202 raft_consensus.cc:359] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.822608 1202 raft_consensus.cc:385] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.822656 1202 raft_consensus.cc:740] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.822978 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 98105bceda9b44bf893f82a4234a3c81
I20251214 20:52:26.824302 1203 tablet_bootstrap.cc:654] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.825407 746 tablet_service.cc:1552] Processing DeleteTablet for tablet ca57963bcfb54eec9fcbed82757d8387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.825541 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 9d89ee602c524c96a21c0a235af07481 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e11723c042a4637839f1b13bed0935a) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.825755 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 16816fa6b8c34207bb929147910b93cb with delete_type TABLET_DATA_DELETED (replaced by tablet 532c44785f204f988518c6fd628f76ca) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.825896 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d6e7306254044449f1127fb963ca5f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.826078 743 tablet_service.cc:1552] Processing DeleteTablet for tablet ca263a80702f4f64aa7c872c2be08207 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:26.826094 447 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:46855 (request call id 175) took 1400 ms (client timeout 1000 ms). Trace:
W20251214 20:52:26.826227 447 rpcz_store.cc:269] 1214 20:52:25.425154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.492141 (+ 66987us) service_pool.cc:225] Handling call
1214 20:52:25.501119 (+ 8978us) spinlock_profiling.cc:243] Waited 7.74 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b17aa08 00007ff411c87ca6 00007ff411c879d5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:26.826024 (+1324905us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:26.050880 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:26.050981 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:26.050986 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:26.050989 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:26.051096 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:26.051106 (+ 10us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:26.051110 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:26.051298 (+ 188us) tablet.cc:699] Row locks acquired
1214 20:52:26.051301 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:26.051337 (+ 36us) write_op.cc:270] Start()
1214 20:52:26.051368 (+ 31us) write_op.cc:276] Timestamp: P: 1765745546051247 usec, L: 0
1214 20:52:26.051371 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:26.051552 (+ 181us) log.cc:844] Serialized 901 byte log entry
1214 20:52:26.207987 (+156435us) op_driver.cc:464] REPLICATION: finished
1214 20:52:26.209199 (+ 1212us) write_op.cc:301] APPLY: starting
1214 20:52:26.209208 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:26.209234 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:26.209262 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:26.209316 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:26.209340 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:26.209346 (+ 6us) write_op.cc:312] APPLY: finished
1214 20:52:26.209395 (+ 49us) log.cc:844] Serialized 29 byte log entry
1214 20:52:26.209431 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.209443 (+ 12us) write_op.cc:454] Released schema lock
1214 20:52:26.209448 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:26.209506 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:26.209509 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"spinlock_wait_cycles":17802624,"child_traces":[["op",{"apply.queue_time_us":1183,"apply.run_cpu_time_us":299,"apply.run_wall_time_us":298,"num_ops":2,"prepare.queue_time_us":54569,"prepare.run_cpu_time_us":767,"prepare.run_wall_time_us":1045,"raft.queue_time_us":34,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":156600}]]}
I20251214 20:52:26.827175 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 98105bceda9b44bf893f82a4234a3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:26.830720 1982 ts_tablet_manager.cc:1929] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.830828 1982 log.cc:1199] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/02e256d4537249a394b80dc36beefef4
I20251214 20:52:26.835577 1202 consensus_queue.cc:260] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.843967 887 tablet_service.cc:1552] Processing DeleteTablet for tablet ca57963bcfb54eec9fcbed82757d8387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.844254 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 98105bceda9b44bf893f82a4234a3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.845890 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d164747cb294bd4b126b4420231867d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:26.850469 989 consensus_peers.cc:597] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:26.851104 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 5da9deeedce0421c9aeaebd71ba11b1b with delete_type TABLET_DATA_DELETED (replaced by tablet 1dca369dc5f54a4ea1783e334c5670d6) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.851104 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet ca57963bcfb54eec9fcbed82757d8387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.851359 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 762068d560ff47918dd9fd8791083433 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.851547 879 tablet_service.cc:1552] Processing DeleteTablet for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.851579 2167 raft_consensus.cc:493] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:26.851780 2167 raft_consensus.cc:515] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.852552 2167 leader_election.cc:290] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:26.852928 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe0c3c8a1d5b46cfacd35d3f0d82e432" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:26.853454 989 leader_election.cc:343] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.853200 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe0c3c8a1d5b46cfacd35d3f0d82e432" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:26.853909 903 raft_consensus.cc:2468] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:26.854274 991 leader_election.cc:304] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:26.854523 2167 raft_consensus.cc:2804] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:26.854590 2167 raft_consensus.cc:493] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:26.854648 2167 raft_consensus.cc:3060] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.853072 1202 ts_tablet_manager.cc:1434] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20251214 20:52:26.855279 1203 tablet_bootstrap.cc:492] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:26.855485 1203 ts_tablet_manager.cc:1403] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.009s
I20251214 20:52:26.856391 1203 raft_consensus.cc:359] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.856568 1203 raft_consensus.cc:385] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.856616 1203 raft_consensus.cc:740] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.856822 1203 consensus_queue.cc:260] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.857254 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 16816fa6b8c34207bb929147910b93cb with delete_type TABLET_DATA_DELETED (replaced by tablet 532c44785f204f988518c6fd628f76ca) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.857326 1203 ts_tablet_manager.cc:1434] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:26.857599 1203 tablet_bootstrap.cc:492] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:26.858872 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f4760585baf4b4ab3b9b94b5a748a61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.859256 1202 tablet_bootstrap.cc:492] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:26.859071 2167 raft_consensus.cc:515] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.859613 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 02e256d4537249a394b80dc36beefef4 with delete_type TABLET_DATA_DELETED (replaced by tablet cea9423c40044e4cabb8bbd755a46688) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.860934 1202 tablet_bootstrap.cc:654] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.861130 1982 ts_tablet_manager.cc:1950] T 02e256d4537249a394b80dc36beefef4 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:26.862440 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe0c3c8a1d5b46cfacd35d3f0d82e432" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:26.862234 2167 leader_election.cc:290] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:26.859465 1203 tablet_bootstrap.cc:654] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:26.862915 989 leader_election.cc:343] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:26.863602 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 2accf8a160584c82bb695a52e3f2b05c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.864337 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 217) took 4471 ms. Trace:
I20251214 20:52:26.863682 1856 ts_tablet_manager.cc:1929] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.869378 1982 rpcz_store.cc:276] 1214 20:52:22.392569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.392670 (+ 101us) service_pool.cc:225] Handling call
1214 20:52:26.747318 (+4354648us) ts_tablet_manager.cc:1151] Deleting tablet 02e256d4537249a394b80dc36beefef4
1214 20:52:26.747327 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.816710 (+ 69383us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.830607 (+ 13897us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.864278 (+ 33671us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4354367}
I20251214 20:52:26.869771 1856 log.cc:1199] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/a38e82f5dd1442f3a1dfcd6fbd94dbaf
I20251214 20:52:26.870221 1856 ts_tablet_manager.cc:1950] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.862600 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe0c3c8a1d5b46cfacd35d3f0d82e432" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:26.870457 903 raft_consensus.cc:3060] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.873214 903 raft_consensus.cc:2468] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:26.863925 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 35a529702f924a5c9bf53c3c668c6094 with delete_type TABLET_DATA_DELETED (replaced by tablet 719d157ac8f74eb7911df05ab5e4a3d5) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:26.874732 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 02e256d4537249a394b80dc36beefef4 (table default.Table_78f54ddbc4d7430fa694c4ce272be743 [id=ed8d16aa8d984c4198c1d1b4ae3fc729]) successfully deleted
I20251214 20:52:26.875141 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 236) took 4214 ms. Trace:
I20251214 20:52:26.875720 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 298) took 3478 ms. Trace:
I20251214 20:52:26.875897 1982 rpcz_store.cc:276] 1214 20:52:22.660308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.664234 (+ 3926us) service_pool.cc:225] Handling call
1214 20:52:26.874954 (+4210720us) ts_tablet_manager.cc:1151] Deleting tablet 6eca34627bbc4a3f9915be72f3128370
1214 20:52:26.874969 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.875103 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4210467}
W20251214 20:52:26.876502 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 6eca34627bbc4a3f9915be72f3128370 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6eca34627bbc4a3f9915be72f3128370 already in progress: creating tablet
I20251214 20:52:26.876538 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 235) took 4216 ms. Trace:
I20251214 20:52:26.876614 1982 rpcz_store.cc:276] 1214 20:52:22.660219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.667195 (+ 6976us) service_pool.cc:225] Handling call
1214 20:52:26.876425 (+4209230us) ts_tablet_manager.cc:1151] Deleting tablet 58ac06f852fc4abea5cb213926002ca1
1214 20:52:26.876436 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.876524 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4208987}
I20251214 20:52:26.876824 1856 rpcz_store.cc:276] 1214 20:52:23.396957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.397014 (+ 57us) service_pool.cc:225] Handling call
1214 20:52:26.778863 (+3381849us) ts_tablet_manager.cc:1151] Deleting tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf
1214 20:52:26.778872 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.812082 (+ 33210us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.863560 (+ 51478us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.875703 (+ 12143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3381605}
I20251214 20:52:26.877480 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf (table default.Table_08488285c9094a9dacc1e489c50271b3 [id=d3b4eef101ec491485c1889f08de9992]) successfully deleted
I20251214 20:52:26.877985 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 299) took 3479 ms. Trace:
I20251214 20:52:26.878414 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 242) took 3975 ms. Trace:
I20251214 20:52:26.868940 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet ec85857f047e44d78835305ddcb70768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.873775 991 leader_election.cc:304] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:26.879230 2167 raft_consensus.cc:2804] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:26.879372 2167 raft_consensus.cc:697] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:26.879630 2167 consensus_queue.cc:237] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.880932 1982 rpcz_store.cc:276] 1214 20:52:22.903067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.906793 (+ 3726us) service_pool.cc:225] Handling call
1214 20:52:26.878334 (+3971541us) ts_tablet_manager.cc:1151] Deleting tablet 4b86f4adaf4540f48a4f42bc45e259e5
1214 20:52:26.878344 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.878400 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3971333}
I20251214 20:52:26.878553 1856 rpcz_store.cc:276] 1214 20:52:23.398517 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.398572 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:26.877858 (+3479286us) ts_tablet_manager.cc:1151] Deleting tablet d01db63fd5d2474a9defdce0b98fc29d
1214 20:52:26.877869 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.877968 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3479094}
W20251214 20:52:26.881608 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet d01db63fd5d2474a9defdce0b98fc29d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d01db63fd5d2474a9defdce0b98fc29d already in progress: creating tablet
I20251214 20:52:26.882089 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 300) took 3482 ms. Trace:
W20251214 20:52:26.882534 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 4b86f4adaf4540f48a4f42bc45e259e5 because the tablet was not found. No further retry: Not found: Tablet not found: 4b86f4adaf4540f48a4f42bc45e259e5
I20251214 20:52:26.882619 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 241) took 3979 ms. Trace:
I20251214 20:52:26.882699 1982 rpcz_store.cc:276] 1214 20:52:22.902992 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.907106 (+ 4114us) service_pool.cc:225] Handling call
1214 20:52:26.882492 (+3975386us) ts_tablet_manager.cc:1151] Deleting tablet f342dafae1d04be3ba38f7c0e4f5d058
1214 20:52:26.882502 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.882603 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3975222}
I20251214 20:52:26.883006 1856 rpcz_store.cc:276] 1214 20:52:23.399474 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.399529 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:26.881957 (+3482428us) ts_tablet_manager.cc:1151] Deleting tablet ae948ffc81764692b92045f5befe0184
1214 20:52:26.881967 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.882070 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3482225}
W20251214 20:52:26.883572 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ae948ffc81764692b92045f5befe0184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae948ffc81764692b92045f5befe0184 already in progress: creating tablet
W20251214 20:52:26.883843 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f342dafae1d04be3ba38f7c0e4f5d058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f342dafae1d04be3ba38f7c0e4f5d058 already in progress: creating tablet
I20251214 20:52:26.884043 1982 tablet_replica.cc:333] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:26.884303 1982 raft_consensus.cc:2243] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.884532 1982 raft_consensus.cc:2272] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.885160 1982 ts_tablet_manager.cc:1916] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.891482 1856 tablet_replica.cc:333] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:26.891866 1856 raft_consensus.cc:2243] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.892359 1856 raft_consensus.cc:2272] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.893064 1856 ts_tablet_manager.cc:1916] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.896854 1202 tablet_bootstrap.cc:492] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:26.897205 1202 ts_tablet_manager.cc:1403] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20251214 20:52:26.898869 1202 raft_consensus.cc:359] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.899267 1202 raft_consensus.cc:385] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.899410 1202 raft_consensus.cc:740] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.899498 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_005b061b84dc4e7aa3fe295da6b99cd1" } modify_external_catalogs: true
I20251214 20:52:26.899783 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_005b061b84dc4e7aa3fe295da6b99cd1" } modify_external_catalogs: true
I20251214 20:52:26.899878 1202 consensus_queue.cc:260] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.900700 1202 ts_tablet_manager.cc:1434] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:26.901549 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_99e702b2f3fc436db82c793284a1670c" } new_table_name: "default.Table_3bc13ee5c12f414c8375152aafd7c0d0" modify_external_catalogs: true
I20251214 20:52:26.908733 454 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:26.908782 1982 ts_tablet_manager.cc:1929] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:26.908887 1982 log.cc:1199] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/5f4760585baf4b4ab3b9b94b5a748a61
I20251214 20:52:26.909180 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_baed953978784a8eb83ffe465a406dfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:26.910229 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baed953978784a8eb83ffe465a406dfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.909583 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d2beb8dc1b054cada5e4c2ec9fabc8d4" } new_table_name: "default.Table_f1e9c435093240e4b1c1eb95635c8e66" modify_external_catalogs: true
I20251214 20:52:26.913983 454 catalog_manager.cc:5654] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:26.918169 1982 ts_tablet_manager.cc:1950] T 5f4760585baf4b4ab3b9b94b5a748a61 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:26.918990 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 239) took 4016 ms. Trace:
I20251214 20:52:26.919085 1982 rpcz_store.cc:276] 1214 20:52:22.902805 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.907379 (+ 4574us) service_pool.cc:225] Handling call
1214 20:52:26.883990 (+3976611us) ts_tablet_manager.cc:1151] Deleting tablet 5f4760585baf4b4ab3b9b94b5a748a61
1214 20:52:26.884000 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.895050 (+ 11050us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.908669 (+ 13619us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.918966 (+ 10297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3976454}
I20251214 20:52:26.919624 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 5f4760585baf4b4ab3b9b94b5a748a61 (table default.Table_c3022f9c5abd44f6b1aa6c05e6361b8e [id=3aaa67725e8442fdbae975274c8596c4]) successfully deleted
I20251214 20:52:26.919737 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 238) took 4017 ms. Trace:
I20251214 20:52:26.919806 1982 rpcz_store.cc:276] 1214 20:52:22.902712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.907634 (+ 4922us) service_pool.cc:225] Handling call
1214 20:52:26.919597 (+4011963us) ts_tablet_manager.cc:1151] Deleting tablet 3b47ca1a03944cf1847d3578856dafa7
1214 20:52:26.919607 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.919723 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4011829}
W20251214 20:52:26.920159 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3b47ca1a03944cf1847d3578856dafa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b47ca1a03944cf1847d3578856dafa7 already in progress: creating tablet
I20251214 20:52:26.920264 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 237) took 4017 ms. Trace:
I20251214 20:52:26.920341 1982 rpcz_store.cc:276] 1214 20:52:22.902607 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.907862 (+ 5255us) service_pool.cc:225] Handling call
1214 20:52:26.920152 (+4012290us) ts_tablet_manager.cc:1151] Deleting tablet 274185a4115b458e8e9a376bdbcc870b
1214 20:52:26.920162 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.920249 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4012135}
W20251214 20:52:26.920701 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 274185a4115b458e8e9a376bdbcc870b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 274185a4115b458e8e9a376bdbcc870b already in progress: creating tablet
I20251214 20:52:26.920796 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 240) took 4017 ms. Trace:
I20251214 20:52:26.920869 1982 rpcz_store.cc:276] 1214 20:52:22.902910 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.908063 (+ 5153us) service_pool.cc:225] Handling call
1214 20:52:26.920700 (+4012637us) ts_tablet_manager.cc:1151] Deleting tablet bc73d590531c409e9f60670435dc8223
1214 20:52:26.920708 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.920782 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4012528}
W20251214 20:52:26.921171 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet bc73d590531c409e9f60670435dc8223 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc73d590531c409e9f60670435dc8223 already in progress: creating tablet
I20251214 20:52:26.921274 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 244) took 3981 ms. Trace:
I20251214 20:52:26.921344 1982 rpcz_store.cc:276] 1214 20:52:22.939519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.939781 (+ 262us) service_pool.cc:225] Handling call
1214 20:52:26.921172 (+3981391us) ts_tablet_manager.cc:1151] Deleting tablet 9b1a937150ce4ecfa210c8465d682496
1214 20:52:26.921181 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.921261 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3981070}
I20251214 20:52:26.921733 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 246) took 3972 ms. Trace:
I20251214 20:52:26.921797 1982 rpcz_store.cc:276] 1214 20:52:22.949642 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:22.949707 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:26.921621 (+3971914us) ts_tablet_manager.cc:1151] Deleting tablet eb29fd8c85344ed8aa8d880e2ed240bc
1214 20:52:26.921629 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.921720 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3971592}
W20251214 20:52:26.922086 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet eb29fd8c85344ed8aa8d880e2ed240bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb29fd8c85344ed8aa8d880e2ed240bc already in progress: creating tablet
I20251214 20:52:26.922113 1982 tablet_replica.cc:333] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:26.922976 1982 raft_consensus.cc:2243] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.923228 1982 raft_consensus.cc:2272] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.923807 1982 ts_tablet_manager.cc:1916] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.930006 1982 ts_tablet_manager.cc:1929] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:26.930124 1982 log.cc:1199] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/023299180d074893847e84dcd9812751
I20251214 20:52:26.930465 1982 ts_tablet_manager.cc:1950] T 023299180d074893847e84dcd9812751 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:26.931267 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 255) took 3594 ms. Trace:
I20251214 20:52:26.931365 1982 rpcz_store.cc:276] 1214 20:52:23.336407 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.336709 (+ 302us) service_pool.cc:225] Handling call
1214 20:52:26.922070 (+3585361us) ts_tablet_manager.cc:1151] Deleting tablet 023299180d074893847e84dcd9812751
1214 20:52:26.922078 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.927211 (+ 5133us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.929899 (+ 2688us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.931245 (+ 1346us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":679,"tablet-delete.queue_time_us":3585102}
I20251214 20:52:26.931813 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 258) took 3522 ms. Trace:
I20251214 20:52:26.931883 1982 rpcz_store.cc:276] 1214 20:52:23.409686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.409742 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:26.931668 (+3521926us) ts_tablet_manager.cc:1151] Deleting tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf
1214 20:52:26.931677 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.931799 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3521659}
I20251214 20:52:26.932204 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 257) took 3522 ms. Trace:
I20251214 20:52:26.932274 1982 rpcz_store.cc:276] 1214 20:52:23.409587 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.409976 (+ 389us) service_pool.cc:225] Handling call
1214 20:52:26.932085 (+3522109us) ts_tablet_manager.cc:1151] Deleting tablet d01db63fd5d2474a9defdce0b98fc29d
1214 20:52:26.932093 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.932189 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3521830}
I20251214 20:52:26.932567 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 259) took 3522 ms. Trace:
I20251214 20:52:26.932637 1982 rpcz_store.cc:276] 1214 20:52:23.409816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.410032 (+ 216us) service_pool.cc:225] Handling call
1214 20:52:26.932478 (+3522446us) ts_tablet_manager.cc:1151] Deleting tablet ae948ffc81764692b92045f5befe0184
1214 20:52:26.932487 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.932554 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3522032}
I20251214 20:52:26.932747 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 023299180d074893847e84dcd9812751 (table default.Table_8ccdf53e3ef149e1a70a20de6caedafa [id=c0729ac770394d7b87ef36e53f85c151]) successfully deleted
I20251214 20:52:26.932866 1982 tablet_replica.cc:333] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:26.933058 1982 raft_consensus.cc:2243] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:26.933161 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf already in progress: creating tablet
I20251214 20:52:26.933219 1982 raft_consensus.cc:2272] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:26.933377 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet d01db63fd5d2474a9defdce0b98fc29d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d01db63fd5d2474a9defdce0b98fc29d already in progress: creating tablet
W20251214 20:52:26.933616 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ae948ffc81764692b92045f5befe0184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae948ffc81764692b92045f5befe0184 already in progress: creating tablet
I20251214 20:52:26.933738 1982 ts_tablet_manager.cc:1916] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.939839 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641a9073c73d41c09e07cabc60735ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.940534 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2bea617954c475bac7e9b8719faa028. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:26.908265 1203 tablet_bootstrap.cc:492] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:26.950063 1203 ts_tablet_manager.cc:1403] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.092s user 0.015s sys 0.018s
I20251214 20:52:26.950292 1977 ts_tablet_manager.cc:1929] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.950412 1977 log.cc:1199] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/3bd5c18696f54ea190c1ef84b93c4f63
I20251214 20:52:26.950734 1977 ts_tablet_manager.cc:1950] T 3bd5c18696f54ea190c1ef84b93c4f63 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:26.953725 1203 raft_consensus.cc:359] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.954120 1203 raft_consensus.cc:385] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.954187 1203 raft_consensus.cc:740] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.954325 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c9b7a105aebf4ed48e4162d811977686" } modify_external_catalogs: true
I20251214 20:52:26.954404 1203 consensus_queue.cc:260] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.954557 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c9b7a105aebf4ed48e4162d811977686" } modify_external_catalogs: true
I20251214 20:52:26.954847 1203 ts_tablet_manager.cc:1434] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:26.941179 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 98105bceda9b44bf893f82a4234a3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:26.955808 1206 tablet_bootstrap.cc:492] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:26.955965 1206 ts_tablet_manager.cc:1403] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.189s user 0.019s sys 0.030s
I20251214 20:52:26.956583 1982 ts_tablet_manager.cc:1929] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:26.956697 1982 log.cc:1199] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/2b92b6d5dde2465684d19aee44a83160
I20251214 20:52:26.956820 1206 raft_consensus.cc:359] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.956986 1206 raft_consensus.cc:385] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.957036 1206 raft_consensus.cc:740] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.957051 1982 ts_tablet_manager.cc:1950] T 2b92b6d5dde2465684d19aee44a83160 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:26.957816 1206 consensus_queue.cc:260] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.958503 1206 ts_tablet_manager.cc:1434] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:26.958791 1206 tablet_bootstrap.cc:492] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:26.960632 1206 tablet_bootstrap.cc:654] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.963089 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_02304abff69e46f68c7152bfc33cb44d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:26.963830 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02304abff69e46f68c7152bfc33cb44d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.964061 1203 tablet_bootstrap.cc:492] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:26.965782 1203 tablet_bootstrap.cc:654] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.966802 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 260) took 3423 ms. Trace:
I20251214 20:52:26.966902 1982 rpcz_store.cc:276] 1214 20:52:23.543475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.543562 (+ 87us) service_pool.cc:225] Handling call
1214 20:52:26.932825 (+3389263us) ts_tablet_manager.cc:1151] Deleting tablet 2b92b6d5dde2465684d19aee44a83160
1214 20:52:26.932834 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.945028 (+ 12194us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.956467 (+ 11439us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.966781 (+ 10314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3389059}
I20251214 20:52:26.967365 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 265) took 3382 ms. Trace:
I20251214 20:52:26.967445 1982 rpcz_store.cc:276] 1214 20:52:23.584634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.584884 (+ 250us) service_pool.cc:225] Handling call
1214 20:52:26.967247 (+3382363us) ts_tablet_manager.cc:1151] Deleting tablet 37a33cc9bb784f72956ac9f5493a0c3d
1214 20:52:26.967257 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.967350 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3382165}
I20251214 20:52:26.967728 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 271) took 3294 ms. Trace:
I20251214 20:52:26.967792 1982 rpcz_store.cc:276] 1214 20:52:23.672818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.676056 (+ 3238us) service_pool.cc:225] Handling call
1214 20:52:26.967620 (+3291564us) ts_tablet_manager.cc:1151] Deleting tablet fc7353593bda4fceaa389c393ddc19c6
1214 20:52:26.967628 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.967713 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3291366}
I20251214 20:52:26.968092 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 275) took 3216 ms. Trace:
I20251214 20:52:26.968148 1982 rpcz_store.cc:276] 1214 20:52:23.751268 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.751327 (+ 59us) service_pool.cc:225] Handling call
1214 20:52:26.967980 (+3216653us) ts_tablet_manager.cc:1151] Deleting tablet 880a78b924574181bdd804d599f4f579
1214 20:52:26.967988 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.968079 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3216228}
I20251214 20:52:26.968428 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 278) took 3194 ms. Trace:
W20251214 20:52:26.968498 452 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9f6da5d29294f08a86c96c6821dda10 consensus state has no leader
I20251214 20:52:26.968562 1982 rpcz_store.cc:276] 1214 20:52:23.773908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.773985 (+ 77us) service_pool.cc:225] Handling call
1214 20:52:26.968315 (+3194330us) ts_tablet_manager.cc:1151] Deleting tablet e68bf6b1a2974957adae70449334df64
1214 20:52:26.968322 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.968413 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3182333}
I20251214 20:52:26.968880 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 282) took 3127 ms. Trace:
I20251214 20:52:26.968948 1982 rpcz_store.cc:276] 1214 20:52:23.841541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.842777 (+ 1236us) service_pool.cc:225] Handling call
1214 20:52:26.968762 (+3125985us) ts_tablet_manager.cc:1151] Deleting tablet 825baba561ee4a8ebd44f75d8c51706a
1214 20:52:26.968770 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.968864 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3125673}
I20251214 20:52:26.969165 1982 tablet_replica.cc:333] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:26.969336 1982 raft_consensus.cc:2243] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.969511 1982 raft_consensus.cc:2272] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.970067 1982 ts_tablet_manager.cc:1916] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.978293 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 2b92b6d5dde2465684d19aee44a83160 (table default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]) successfully deleted
W20251214 20:52:26.978644 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 37a33cc9bb784f72956ac9f5493a0c3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37a33cc9bb784f72956ac9f5493a0c3d already in progress: creating tablet
W20251214 20:52:26.978817 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet fc7353593bda4fceaa389c393ddc19c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc7353593bda4fceaa389c393ddc19c6 already in progress: creating tablet
W20251214 20:52:26.978974 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 880a78b924574181bdd804d599f4f579 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 880a78b924574181bdd804d599f4f579 already in progress: creating tablet
W20251214 20:52:26.979113 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet e68bf6b1a2974957adae70449334df64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e68bf6b1a2974957adae70449334df64 already in progress: creating tablet
W20251214 20:52:26.979383 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 825baba561ee4a8ebd44f75d8c51706a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 825baba561ee4a8ebd44f75d8c51706a already in progress: creating tablet
I20251214 20:52:26.979820 1982 ts_tablet_manager.cc:1929] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:26.979940 1982 log.cc:1199] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/3bd5c18696f54ea190c1ef84b93c4f63
I20251214 20:52:26.980279 1982 ts_tablet_manager.cc:1950] T 3bd5c18696f54ea190c1ef84b93c4f63 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:26.980093 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_7219830d8fb040debc2388f49091b802"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:26.980574 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 283) took 3110 ms. Trace:
I20251214 20:52:26.980645 1982 rpcz_store.cc:276] 1214 20:52:23.869912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.869999 (+ 87us) service_pool.cc:225] Handling call
1214 20:52:26.969127 (+3099128us) ts_tablet_manager.cc:1151] Deleting tablet 3bd5c18696f54ea190c1ef84b93c4f63
1214 20:52:26.969135 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.976873 (+ 7738us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.979718 (+ 2845us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.980559 (+ 841us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3098973}
W20251214 20:52:26.980857 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7219830d8fb040debc2388f49091b802 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:26.980983 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 284) took 3101 ms. Trace:
I20251214 20:52:26.981050 1982 rpcz_store.cc:276] 1214 20:52:23.879883 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.880343 (+ 460us) service_pool.cc:225] Handling call
1214 20:52:26.980901 (+3100558us) ts_tablet_manager.cc:1151] Deleting tablet e16fe751f10f4616a00d0af811998729
1214 20:52:26.980909 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.980970 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3100378}
I20251214 20:52:26.981271 1982 tablet_replica.cc:333] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:26.981451 1982 raft_consensus.cc:2243] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:26.981765 1982 raft_consensus.cc:2272] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.982115 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]) successfully deleted
W20251214 20:52:26.982396 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet e16fe751f10f4616a00d0af811998729 because the tablet was not found. No further retry: Not found: Tablet not found: e16fe751f10f4616a00d0af811998729
I20251214 20:52:26.986773 1202 tablet_bootstrap.cc:492] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:26.989341 1206 tablet_bootstrap.cc:492] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:26.989498 1206 ts_tablet_manager.cc:1403] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20251214 20:52:26.990263 1206 raft_consensus.cc:359] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.990435 1206 raft_consensus.cc:385] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:26.990483 1206 raft_consensus.cc:740] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:26.990741 1206 consensus_queue.cc:260] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:26.991418 1206 ts_tablet_manager.cc:1434] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:26.991700 1206 tablet_bootstrap.cc:492] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:26.991827 1856 ts_tablet_manager.cc:1929] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:26.991935 1856 log.cc:1199] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/2b92b6d5dde2465684d19aee44a83160
I20251214 20:52:26.992010 1202 tablet_bootstrap.cc:654] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.992280 1856 ts_tablet_manager.cc:1950] T 2b92b6d5dde2465684d19aee44a83160 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:26.993109 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 301) took 3448 ms. Trace:
I20251214 20:52:26.993345 1206 tablet_bootstrap.cc:654] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:26.993204 1856 rpcz_store.cc:276] 1214 20:52:23.544796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.544868 (+ 72us) service_pool.cc:225] Handling call
1214 20:52:26.883892 (+3339024us) ts_tablet_manager.cc:1151] Deleting tablet 2b92b6d5dde2465684d19aee44a83160
1214 20:52:26.891402 (+ 7510us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.988576 (+ 97174us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.991716 (+ 3140us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.993085 (+ 1369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3337840}
I20251214 20:52:26.993559 1856 tablet_replica.cc:333] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:26.993537 768 raft_consensus.cc:3060] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:26.993763 1856 raft_consensus.cc:2243] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:26.993899 1856 raft_consensus.cc:2272] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:26.995752 1982 ts_tablet_manager.cc:1916] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:26.999534 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 2b92b6d5dde2465684d19aee44a83160 (table default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]) successfully deleted
I20251214 20:52:27.003638 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 492770a7b8b74ba49315f974cc54eedb
I20251214 20:52:27.004647 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_1f214ce7548c43af80a74e233714ca0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.005378 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f214ce7548c43af80a74e233714ca0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.006846 1982 ts_tablet_manager.cc:1929] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.006991 1982 log.cc:1199] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/762068d560ff47918dd9fd8791083433
I20251214 20:52:27.010843 1982 ts_tablet_manager.cc:1950] T 762068d560ff47918dd9fd8791083433 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:27.011229 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 285) took 3106 ms. Trace:
I20251214 20:52:27.011307 1982 rpcz_store.cc:276] 1214 20:52:23.905013 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.905222 (+ 209us) service_pool.cc:225] Handling call
1214 20:52:26.981231 (+3076009us) ts_tablet_manager.cc:1151] Deleting tablet 762068d560ff47918dd9fd8791083433
1214 20:52:26.981238 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.998879 (+ 17641us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.006715 (+ 7836us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.011209 (+ 4494us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3075823}
I20251214 20:52:27.011710 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 286) took 3096 ms. Trace:
I20251214 20:52:27.011775 1982 rpcz_store.cc:276] 1214 20:52:23.915410 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.915468 (+ 58us) service_pool.cc:225] Handling call
1214 20:52:27.011589 (+3096121us) ts_tablet_manager.cc:1151] Deleting tablet b69bb4d5979f492683fc13238a668b9f
1214 20:52:27.011597 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.011694 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3095917}
I20251214 20:52:27.012121 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 287) took 3089 ms. Trace:
I20251214 20:52:27.012187 1982 rpcz_store.cc:276] 1214 20:52:23.922394 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.922525 (+ 131us) service_pool.cc:225] Handling call
1214 20:52:27.012007 (+3089482us) ts_tablet_manager.cc:1151] Deleting tablet fe0c3c8a1d5b46cfacd35d3f0d82e432
1214 20:52:27.012015 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.012107 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3089299}
I20251214 20:52:27.012485 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 288) took 3090 ms. Trace:
I20251214 20:52:27.012555 1982 rpcz_store.cc:276] 1214 20:52:23.922511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.922744 (+ 233us) service_pool.cc:225] Handling call
1214 20:52:27.012397 (+3089653us) ts_tablet_manager.cc:1151] Deleting tablet 34df3a34b15b4fe19bd12f785b07a033
1214 20:52:27.012405 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.012470 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3089508}
W20251214 20:52:27.012652 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f9def8e4fd449599820a63845397750 consensus state has no leader
I20251214 20:52:27.012820 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 298) took 2982 ms. Trace:
I20251214 20:52:27.012885 1982 rpcz_store.cc:276] 1214 20:52:24.029847 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.029892 (+ 45us) service_pool.cc:225] Handling call
1214 20:52:27.012725 (+2982833us) ts_tablet_manager.cc:1151] Deleting tablet 1d7aa508b11944aa8e268741c1998a9e
1214 20:52:27.012733 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.012808 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2982634}
I20251214 20:52:27.013123 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 297) took 2983 ms. Trace:
I20251214 20:52:27.013181 1982 rpcz_store.cc:276] 1214 20:52:24.029773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.034624 (+ 4851us) service_pool.cc:225] Handling call
1214 20:52:27.013039 (+2978415us) ts_tablet_manager.cc:1151] Deleting tablet 3bed2c326906486e99300e05033c9c0f
1214 20:52:27.013046 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.013112 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2978096}
I20251214 20:52:27.013435 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 304) took 2838 ms. Trace:
I20251214 20:52:27.013491 1982 rpcz_store.cc:276] 1214 20:52:24.174996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.175064 (+ 68us) service_pool.cc:225] Handling call
1214 20:52:27.013336 (+2838272us) ts_tablet_manager.cc:1151] Deleting tablet feb19ea389af4ea6bb158a0e8aaeb094
1214 20:52:27.013343 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.013423 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2837908}
I20251214 20:52:27.013758 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 307) took 2710 ms. Trace:
I20251214 20:52:27.013819 1982 rpcz_store.cc:276] 1214 20:52:24.303788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.303860 (+ 72us) service_pool.cc:225] Handling call
1214 20:52:27.013655 (+2709795us) ts_tablet_manager.cc:1151] Deleting tablet 43fe3bb1ed3941ce9078376f8f478408
1214 20:52:27.013661 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.013743 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2709539}
I20251214 20:52:27.014114 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 308) took 2707 ms. Trace:
I20251214 20:52:27.014180 1982 rpcz_store.cc:276] 1214 20:52:24.307075 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.307332 (+ 257us) service_pool.cc:225] Handling call
1214 20:52:27.014005 (+2706673us) ts_tablet_manager.cc:1151] Deleting tablet 054cfd587d19409fbdf7f4848bcaf3f9
1214 20:52:27.014014 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.014100 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2679834}
I20251214 20:52:27.014412 1982 tablet_replica.cc:333] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:27.014600 1982 raft_consensus.cc:2243] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.014761 1982 raft_consensus.cc:2272] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.015386 1982 ts_tablet_manager.cc:1916] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.015902 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 762068d560ff47918dd9fd8791083433 (table default.Table_431bf74dfb32400bb0b2ef098c18e8df [id=2fcb81bfc6904fdd924de2494aba2e4a]) successfully deleted
I20251214 20:52:27.016148 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 492770a7b8b74ba49315f974cc54eedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:27.016220 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b69bb4d5979f492683fc13238a668b9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b69bb4d5979f492683fc13238a668b9f already in progress: creating tablet
W20251214 20:52:27.016388 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 already in progress: creating tablet
W20251214 20:52:27.016578 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 34df3a34b15b4fe19bd12f785b07a033 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34df3a34b15b4fe19bd12f785b07a033 already in progress: creating tablet
W20251214 20:52:27.016722 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 1d7aa508b11944aa8e268741c1998a9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d7aa508b11944aa8e268741c1998a9e already in progress: creating tablet
W20251214 20:52:27.016876 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3bed2c326906486e99300e05033c9c0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bed2c326906486e99300e05033c9c0f already in progress: creating tablet
W20251214 20:52:27.017040 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet feb19ea389af4ea6bb158a0e8aaeb094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet feb19ea389af4ea6bb158a0e8aaeb094 already in progress: creating tablet
W20251214 20:52:27.017215 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 43fe3bb1ed3941ce9078376f8f478408 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43fe3bb1ed3941ce9078376f8f478408 already in progress: creating tablet
W20251214 20:52:27.017359 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 054cfd587d19409fbdf7f4848bcaf3f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 054cfd587d19409fbdf7f4848bcaf3f9 already in progress: creating tablet
I20251214 20:52:27.018366 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 492770a7b8b74ba49315f974cc54eedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.019865 1206 tablet_bootstrap.cc:492] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:27.020085 1206 ts_tablet_manager.cc:1403] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20251214 20:52:27.021112 1206 raft_consensus.cc:359] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.021359 1206 raft_consensus.cc:385] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.021433 1206 raft_consensus.cc:740] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.021699 1206 consensus_queue.cc:260] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.021792 2468 raft_consensus.cc:493] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.021963 2468 raft_consensus.cc:515] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.022341 1206 ts_tablet_manager.cc:1434] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:27.022656 1206 tablet_bootstrap.cc:492] T f900ebe92efa476cac232131a818d449 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:27.022814 2468 leader_election.cc:290] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:27.027094 775 raft_consensus.cc:3060] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.035395 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408013cf49e46d0b18d24b48483ea49" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:27.040868 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 492770a7b8b74ba49315f974cc54eedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.042605 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4408013cf49e46d0b18d24b48483ea49" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:27.043550 713 leader_election.cc:304] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:27.044092 2470 raft_consensus.cc:2749] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.046054 1856 ts_tablet_manager.cc:1916] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.062844 1856 ts_tablet_manager.cc:1929] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:27.065625 1982 ts_tablet_manager.cc:1929] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.066285 1982 log.cc:1199] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ac64f4fcb6a943668584658df19f950c
I20251214 20:52:27.067261 1856 log.cc:1199] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/37a33cc9bb784f72956ac9f5493a0c3d
I20251214 20:52:27.067867 1856 ts_tablet_manager.cc:1950] T 37a33cc9bb784f72956ac9f5493a0c3d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:27.068778 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 306) took 3486 ms. Trace:
I20251214 20:52:27.068926 1856 rpcz_store.cc:276] 1214 20:52:23.582537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.582677 (+ 140us) service_pool.cc:225] Handling call
1214 20:52:26.993501 (+3410824us) ts_tablet_manager.cc:1151] Deleting tablet 37a33cc9bb784f72956ac9f5493a0c3d
1214 20:52:26.993511 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.053126 (+ 59615us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.062698 (+ 9572us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.068756 (+ 6058us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3410643}
I20251214 20:52:27.069607 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]) successfully deleted
I20251214 20:52:27.069862 1206 tablet_bootstrap.cc:654] T f900ebe92efa476cac232131a818d449 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.070290 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 313) took 3397 ms. Trace:
I20251214 20:52:27.070436 1856 rpcz_store.cc:276] 1214 20:52:23.673120 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.673177 (+ 57us) service_pool.cc:225] Handling call
1214 20:52:27.070139 (+3396962us) ts_tablet_manager.cc:1151] Deleting tablet fc7353593bda4fceaa389c393ddc19c6
1214 20:52:27.070150 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.070274 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3396691}
W20251214 20:52:27.070915 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet fc7353593bda4fceaa389c393ddc19c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc7353593bda4fceaa389c393ddc19c6 already in progress: creating tablet
I20251214 20:52:27.071389 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 316) took 3319 ms. Trace:
I20251214 20:52:27.071511 1856 rpcz_store.cc:276] 1214 20:52:23.751574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.800707 (+ 49133us) service_pool.cc:225] Handling call
1214 20:52:27.071291 (+3270584us) ts_tablet_manager.cc:1151] Deleting tablet 880a78b924574181bdd804d599f4f579
1214 20:52:27.071300 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.071374 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3267940}
W20251214 20:52:27.071952 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 880a78b924574181bdd804d599f4f579 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 880a78b924574181bdd804d599f4f579 already in progress: creating tablet
I20251214 20:52:27.072283 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 319) took 3305 ms. Trace:
I20251214 20:52:27.072387 1856 rpcz_store.cc:276] 1214 20:52:23.766848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.815944 (+ 49096us) service_pool.cc:225] Handling call
1214 20:52:27.072193 (+3256249us) ts_tablet_manager.cc:1151] Deleting tablet e68bf6b1a2974957adae70449334df64
1214 20:52:27.072202 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.072269 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3252901}
W20251214 20:52:27.073035 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet e68bf6b1a2974957adae70449334df64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e68bf6b1a2974957adae70449334df64 already in progress: creating tablet
I20251214 20:52:27.073843 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 325) took 3212 ms. Trace:
I20251214 20:52:27.073947 1856 rpcz_store.cc:276] 1214 20:52:23.861065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.892157 (+ 31092us) service_pool.cc:225] Handling call
1214 20:52:27.073763 (+3181606us) ts_tablet_manager.cc:1151] Deleting tablet 7fcfcfe81fe24d369eea696efc61d21c
1214 20:52:27.073772 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.073826 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3181021}
I20251214 20:52:27.075022 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 280) took 3224 ms. Trace:
W20251214 20:52:27.075174 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 7fcfcfe81fe24d369eea696efc61d21c because the tablet was not found. No further retry: Not found: Tablet not found: 7fcfcfe81fe24d369eea696efc61d21c
I20251214 20:52:27.075502 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 326) took 3213 ms. Trace:
I20251214 20:52:27.075385 1977 rpcz_store.cc:276] 1214 20:52:23.850632 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.916281 (+ 65649us) service_pool.cc:225] Handling call
1214 20:52:26.761003 (+2844722us) ts_tablet_manager.cc:1151] Deleting tablet 3bd5c18696f54ea190c1ef84b93c4f63
1214 20:52:26.761011 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:26.939993 (+178982us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:26.950168 (+ 10175us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.074996 (+124828us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2841080}
I20251214 20:52:27.075628 1856 rpcz_store.cc:276] 1214 20:52:23.861573 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.903206 (+ 41633us) service_pool.cc:225] Handling call
1214 20:52:27.075428 (+3172222us) ts_tablet_manager.cc:1151] Deleting tablet bb698ff30fcc47d09000cc68d2c5225b
1214 20:52:27.075437 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.075488 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3171738}
W20251214 20:52:27.075989 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet bb698ff30fcc47d09000cc68d2c5225b because the tablet was not found. No further retry: Not found: Tablet not found: bb698ff30fcc47d09000cc68d2c5225b
I20251214 20:52:27.076321 1856 tablet_replica.cc:333] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:27.076252 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]) successfully deleted
I20251214 20:52:27.076619 1856 raft_consensus.cc:2243] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:27.076790 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 284) took 3155 ms. Trace:
I20251214 20:52:27.076987 1977 rpcz_store.cc:276] 1214 20:52:23.921380 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.921430 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:27.076635 (+3155205us) ts_tablet_manager.cc:1151] Deleting tablet 34df3a34b15b4fe19bd12f785b07a033
1214 20:52:27.076657 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.076775 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3154939}
W20251214 20:52:27.077653 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 34df3a34b15b4fe19bd12f785b07a033 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34df3a34b15b4fe19bd12f785b07a033 already in progress: creating tablet
I20251214 20:52:27.078130 1977 tablet_replica.cc:333] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:27.078490 1977 raft_consensus.cc:2243] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.078819 1977 raft_consensus.cc:2272] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.079833 1977 ts_tablet_manager.cc:1916] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.084088 1856 raft_consensus.cc:2272] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.084978 1856 ts_tablet_manager.cc:1916] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.086524 1982 ts_tablet_manager.cc:1950] T ac64f4fcb6a943668584658df19f950c P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:27.086879 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 305) took 2903 ms. Trace:
I20251214 20:52:27.091768 1856 ts_tablet_manager.cc:1929] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.092087 1206 tablet_bootstrap.cc:492] T f900ebe92efa476cac232131a818d449 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:27.092288 1206 ts_tablet_manager.cc:1403] T f900ebe92efa476cac232131a818d449 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.004s
I20251214 20:52:27.093187 1206 raft_consensus.cc:359] T f900ebe92efa476cac232131a818d449 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.093406 1206 raft_consensus.cc:385] T f900ebe92efa476cac232131a818d449 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.093479 1206 raft_consensus.cc:740] T f900ebe92efa476cac232131a818d449 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.093722 1206 consensus_queue.cc:260] T f900ebe92efa476cac232131a818d449 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.093972 1856 log.cc:1199] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/3bd5c18696f54ea190c1ef84b93c4f63
I20251214 20:52:27.094424 1206 ts_tablet_manager.cc:1434] T f900ebe92efa476cac232131a818d449 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:27.094713 1206 tablet_bootstrap.cc:492] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:27.094929 1856 ts_tablet_manager.cc:1950] T 3bd5c18696f54ea190c1ef84b93c4f63 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:27.095669 1202 tablet_bootstrap.cc:492] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:27.095994 1202 ts_tablet_manager.cc:1403] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.109s user 0.009s sys 0.004s
I20251214 20:52:27.096951 1202 raft_consensus.cc:359] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.101253 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d5446dd7bdf14b76ab6136efd369dd12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.101974 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5446dd7bdf14b76ab6136efd369dd12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:27.112910 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2688515bd1441fbad7516f8343ca775 consensus state has no leader
I20251214 20:52:27.113395 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a4359e4ce4cb4a5caafc148f1dadef64" } new_table_name: "default.Table_694f6a7aab6f4c76b248dd05c441ea03" modify_external_catalogs: true
I20251214 20:52:27.115095 449 master_service.cc:954] ReplaceTablet: received request to replace tablet 79e621f29c01448399d22615284fccd4 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:27.119016 773 raft_consensus.cc:3060] T f900ebe92efa476cac232131a818d449 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.128609 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 6a000951f61948ac85fdd431f3d162c2
I20251214 20:52:27.130949 450 catalog_manager.cc:5654] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:27.131489 450 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:27.131867 450 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:27.132104 450 catalog_manager.cc:5654] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:27.132474 450 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:27.132790 450 catalog_manager.cc:5654] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:27.135448 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a000951f61948ac85fdd431f3d162c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.138528 448 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_adc1626f17674e3cab49c72818eec7d6 [id=0d173bfba2b343dbac6a02a5ff84c828] alter complete (version 3)
I20251214 20:52:27.112258 1203 tablet_bootstrap.cc:492] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:27.139406 1203 ts_tablet_manager.cc:1403] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.175s user 0.015s sys 0.019s
I20251214 20:52:27.136276 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a000951f61948ac85fdd431f3d162c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.140364 1203 raft_consensus.cc:359] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.140551 1203 raft_consensus.cc:385] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.140604 1203 raft_consensus.cc:740] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.140805 1203 consensus_queue.cc:260] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.141081 1206 tablet_bootstrap.cc:654] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.141316 1203 ts_tablet_manager.cc:1434] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:27.145751 1203 tablet_bootstrap.cc:492] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:27.147815 1203 tablet_bootstrap.cc:654] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.147914 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 37a33cc9bb784f72956ac9f5493a0c3d with delete_type TABLET_DATA_DELETED (replaced by tablet b87b0871c2454971a15dd5643c155694) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.148092 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f with delete_type TABLET_DATA_DELETED (replaced by tablet f11834cabdad445b8e53cd237d146282) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.147753 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 69f6108792fa4c2a9744e3248c76251f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.150010 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 242a77ae57174de48f4eaf43384ee0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.151461 1977 ts_tablet_manager.cc:1929] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:27.151702 1977 log.cc:1199] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/fe0c3c8a1d5b46cfacd35d3f0d82e432
I20251214 20:52:27.152194 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 324) took 3298 ms. Trace:
I20251214 20:52:27.152300 1856 rpcz_store.cc:276] 1214 20:52:23.854106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.855109 (+ 1003us) service_pool.cc:225] Handling call
1214 20:52:27.076272 (+3221163us) ts_tablet_manager.cc:1151] Deleting tablet 3bd5c18696f54ea190c1ef84b93c4f63
1214 20:52:27.076280 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.088576 (+ 12296us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.091658 (+ 3082us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.152173 (+ 60515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3172369}
I20251214 20:52:27.152474 1977 ts_tablet_manager.cc:1950] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:27.152711 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 323) took 3310 ms. Trace:
I20251214 20:52:27.152786 1856 rpcz_store.cc:276] 1214 20:52:23.842701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.843623 (+ 922us) service_pool.cc:225] Handling call
1214 20:52:27.152576 (+3308953us) ts_tablet_manager.cc:1151] Deleting tablet 825baba561ee4a8ebd44f75d8c51706a
1214 20:52:27.152586 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.152696 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3248632}
I20251214 20:52:27.153050 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 327) took 3241 ms. Trace:
I20251214 20:52:27.153126 1856 rpcz_store.cc:276] 1214 20:52:23.911353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.912500 (+ 1147us) service_pool.cc:225] Handling call
1214 20:52:27.152963 (+3240463us) ts_tablet_manager.cc:1151] Deleting tablet b69bb4d5979f492683fc13238a668b9f
1214 20:52:27.152970 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.153037 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3240158}
I20251214 20:52:27.154327 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a000951f61948ac85fdd431f3d162c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.154783 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 012b81223d0f4ae9b742014e329f11a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:17 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.156524 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1]) successfully deleted
W20251214 20:52:27.156821 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 825baba561ee4a8ebd44f75d8c51706a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 825baba561ee4a8ebd44f75d8c51706a already in progress: creating tablet
W20251214 20:52:27.157049 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b69bb4d5979f492683fc13238a668b9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b69bb4d5979f492683fc13238a668b9f already in progress: creating tablet
I20251214 20:52:27.157601 1206 tablet_bootstrap.cc:492] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:27.157756 1206 ts_tablet_manager.cc:1403] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.003s
I20251214 20:52:27.158737 1206 raft_consensus.cc:359] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.158915 1206 raft_consensus.cc:385] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.158954 1206 raft_consensus.cc:740] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.159158 1206 consensus_queue.cc:260] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.159981 1206 ts_tablet_manager.cc:1434] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:27.160123 1103 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:27.160450 1206 tablet_bootstrap.cc:492] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:27.163370 2287 raft_consensus.cc:493] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.163569 2287 raft_consensus.cc:515] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:27.164281 2287 leader_election.cc:290] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:27.168465 1856 tablet_replica.cc:333] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:27.168716 1856 raft_consensus.cc:2243] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:27.168967 1856 pending_rounds.cc:70] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:27.171459 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bdcecba5534294aca1eb419f18e9d2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:27.171686 1050 raft_consensus.cc:2468] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:27.174528 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 283) took 3251 ms. Trace:
I20251214 20:52:27.169006 1856 pending_rounds.cc:77] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493760021422080 op_type: NO_OP noop_request { }
I20251214 20:52:27.175249 1856 raft_consensus.cc:2889] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:27.175359 1856 raft_consensus.cc:2272] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.176913 2523 raft_consensus.cc:493] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.177132 2523 raft_consensus.cc:515] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:27.177246 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_aedc10034f9947a2bf065dda745b96f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.178020 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aedc10034f9947a2bf065dda745b96f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.178056 2523 leader_election.cc:290] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:27.179577 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_dbcc1409fc7e48cb804842a37eebaad2" } modify_external_catalogs: true
I20251214 20:52:27.179811 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:27.175339 1977 rpcz_store.cc:276] 1214 20:52:23.921258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.923270 (+ 2012us) service_pool.cc:225] Handling call
1214 20:52:27.078028 (+3154758us) ts_tablet_manager.cc:1151] Deleting tablet fe0c3c8a1d5b46cfacd35d3f0d82e432
1214 20:52:27.078062 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.130072 (+ 52010us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.151320 (+ 21248us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.172754 (+ 21434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3154512}
I20251214 20:52:27.180107 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_70e73cbd7f9141f3ab482647b6118d52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:27.181300 1203 tablet_bootstrap.cc:492] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
W20251214 20:52:27.181567 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70e73cbd7f9141f3ab482647b6118d52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.182943 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 (table default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]) successfully deleted
I20251214 20:52:27.183585 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fef1d6add764fd28a8f4995b337ea71" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:27.183665 1856 ts_tablet_manager.cc:1916] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.183811 903 raft_consensus.cc:2468] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:27.184950 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a258f399b3e44e009cf7e014db9079aa" } new_table_name: "default.Table_fe0d773656e24c0e9d6e2e36caeaa9f8" modify_external_catalogs: true
I20251214 20:52:27.186240 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_acd41f528acd483cb2c37531c45283d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.186753 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acd41f528acd483cb2c37531c45283d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.189740 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c973e0111804b8b869068bdcc5869bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.190017 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fef1d6add764fd28a8f4995b337ea71" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:27.184309 712 leader_election.cc:304] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:27.181754 1206 tablet_bootstrap.cc:654] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.190757 2523 raft_consensus.cc:2804] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:27.190833 2523 raft_consensus.cc:493] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:27.190891 2523 raft_consensus.cc:3060] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.190814 1203 ts_tablet_manager.cc:1403] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.002s
I20251214 20:52:27.191151 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 292) took 3173 ms. Trace:
I20251214 20:52:27.191672 1977 rpcz_store.cc:276] 1214 20:52:24.017281 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.017350 (+ 69us) service_pool.cc:225] Handling call
1214 20:52:27.181615 (+3164265us) ts_tablet_manager.cc:1151] Deleting tablet 3bed2c326906486e99300e05033c9c0f
1214 20:52:27.181629 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.190884 (+ 9255us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3103747}
I20251214 20:52:27.194648 2523 raft_consensus.cc:515] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:27.194650 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 294) took 3109 ms. Trace:
W20251214 20:52:27.195672 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3bed2c326906486e99300e05033c9c0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bed2c326906486e99300e05033c9c0f already in progress: creating tablet
I20251214 20:52:27.196043 851 leader_election.cc:304] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:27.197001 2287 raft_consensus.cc:2804] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:27.197084 2287 raft_consensus.cc:493] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:27.197142 2287 raft_consensus.cc:3060] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.197969 1982 rpcz_store.cc:276] 1214 20:52:24.183821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.183894 (+ 73us) service_pool.cc:225] Handling call
1214 20:52:27.014369 (+2830475us) ts_tablet_manager.cc:1151] Deleting tablet ac64f4fcb6a943668584658df19f950c
1214 20:52:27.014376 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.018431 (+ 4055us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.056448 (+ 38017us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.086865 (+ 30417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2678434}
I20251214 20:52:27.195940 1977 rpcz_store.cc:276] 1214 20:52:24.085088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.085308 (+ 220us) service_pool.cc:225] Handling call
1214 20:52:27.193606 (+3108298us) ts_tablet_manager.cc:1151] Deleting tablet bc73d590531c409e9f60670435dc8223
1214 20:52:27.193619 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.194621 (+ 1002us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3102759}
I20251214 20:52:27.198505 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ac64f4fcb6a943668584658df19f950c (table default.Table_5275181d46dc4a85b27fc0c45fa587fe [id=ab9838348eb24201bcb2866b16994965]) successfully deleted
I20251214 20:52:27.200448 2167 consensus_queue.cc:1048] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:27.200479 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 298) took 3111 ms. Trace:
I20251214 20:52:27.200771 1977 rpcz_store.cc:276] 1214 20:52:24.089190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.089258 (+ 68us) service_pool.cc:225] Handling call
1214 20:52:27.199530 (+3110272us) ts_tablet_manager.cc:1151] Deleting tablet 3e5e220d083243eb8aa4328f366cd7ff
1214 20:52:27.199714 (+ 184us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.200354 (+ 640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3108476}
I20251214 20:52:27.202636 1202 raft_consensus.cc:385] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.203276 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 314) took 2827 ms. Trace:
W20251214 20:52:27.203387 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet bc73d590531c409e9f60670435dc8223 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc73d590531c409e9f60670435dc8223 already in progress: creating tablet
I20251214 20:52:27.203799 1982 rpcz_store.cc:276] 1214 20:52:24.376119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.376809 (+ 690us) service_pool.cc:225] Handling call
1214 20:52:27.203091 (+2826282us) ts_tablet_manager.cc:1151] Deleting tablet 727e7b61ae3942098712720d344c5310
1214 20:52:27.203105 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.203251 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2826036}
W20251214 20:52:27.204490 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 727e7b61ae3942098712720d344c5310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 727e7b61ae3942098712720d344c5310 already in progress: creating tablet
W20251214 20:52:27.204504 713 leader_election.cc:343] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.205106 2523 leader_election.cc:290] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:27.180923 447 master_service.cc:954] ReplaceTablet: received request to replace tablet 1c5ee4114cb34540af2c5b332ebe82ab from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:27.180871 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_dbcc1409fc7e48cb804842a37eebaad2" } modify_external_catalogs: true
I20251214 20:52:27.205880 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fef1d6add764fd28a8f4995b337ea71" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:27.206386 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 331) took 2480 ms. Trace:
I20251214 20:52:27.206478 1982 rpcz_store.cc:276] 1214 20:52:24.726401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.726635 (+ 234us) service_pool.cc:225] Handling call
1214 20:52:27.206241 (+2479606us) ts_tablet_manager.cc:1151] Deleting tablet 54e5fff1fc84419ab3d3131cb3a12d92
1214 20:52:27.206255 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.206368 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2479328}
I20251214 20:52:27.207420 1982 tablet_replica.cc:333] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:27.207626 1982 raft_consensus.cc:2243] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:27.207747 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 54e5fff1fc84419ab3d3131cb3a12d92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54e5fff1fc84419ab3d3131cb3a12d92 already in progress: creating tablet
I20251214 20:52:27.210286 1982 raft_consensus.cc:2272] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.211303 1982 ts_tablet_manager.cc:1916] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.211220 2455 consensus_queue.cc:1048] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:27.212018 1202 raft_consensus.cc:740] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.204751 962 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:27.205737 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6fef1d6add764fd28a8f4995b337ea71" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:27.213376 904 raft_consensus.cc:3060] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.213928 1202 consensus_queue.cc:260] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.216420 904 raft_consensus.cc:2468] T 6fef1d6add764fd28a8f4995b337ea71 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:27.217711 1202 ts_tablet_manager.cc:1434] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.122s user 0.018s sys 0.000s
I20251214 20:52:27.218286 712 leader_election.cc:304] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:27.218782 2523 raft_consensus.cc:2804] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:27.219287 2523 raft_consensus.cc:697] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:27.219623 2523 consensus_queue.cc:237] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:27.221100 1202 tablet_bootstrap.cc:492] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:27.221143 1206 tablet_bootstrap.cc:492] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:27.221282 1206 ts_tablet_manager.cc:1403] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.004s
I20251214 20:52:27.221966 1206 raft_consensus.cc:359] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.222124 1206 raft_consensus.cc:385] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.222173 1206 raft_consensus.cc:740] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.222363 1206 consensus_queue.cc:260] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.222965 1206 ts_tablet_manager.cc:1434] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:27.223340 1206 tablet_bootstrap.cc:492] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:27.225154 1206 tablet_bootstrap.cc:654] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.231539 1856 ts_tablet_manager.cc:1929] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.237951 1856 log.cc:1199] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/fe0c3c8a1d5b46cfacd35d3f0d82e432
I20251214 20:52:27.240461 1856 ts_tablet_manager.cc:1950] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:27.243443 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 79e621f29c01448399d22615284fccd4
I20251214 20:52:27.244037 449 catalog_manager.cc:6724] ReplaceTablet: tablet 79e621f29c01448399d22615284fccd4 deleted and replaced by tablet 202b1d53cab644c98838b0ee81f29f22
I20251214 20:52:27.244594 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_246962600e464910aa3a756037dd08e9" } modify_external_catalogs: true
I20251214 20:52:27.244872 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_246962600e464910aa3a756037dd08e9" } modify_external_catalogs: true
I20251214 20:52:27.245127 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 328) took 3324 ms. Trace:
I20251214 20:52:27.248227 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e621f29c01448399d22615284fccd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 202b1d53cab644c98838b0ee81f29f22) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:27.248395 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0d0e8f35bf44c9a8797b7507dcce3b9 consensus state has no leader
I20251214 20:52:27.248906 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_78f54ddbc4d7430fa694c4ce272be743" } modify_external_catalogs: true
I20251214 20:52:27.248489 1856 rpcz_store.cc:276] 1214 20:52:23.920720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.920777 (+ 57us) service_pool.cc:225] Handling call
1214 20:52:27.153313 (+3232536us) ts_tablet_manager.cc:1151] Deleting tablet fe0c3c8a1d5b46cfacd35d3f0d82e432
1214 20:52:27.163305 (+ 9992us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.183172 (+ 19867us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x60d0000131c8. stack: 00007fd8605acee6 00007fd8605acca0 00007fd85f5883ad 0000000000984f94 0000000000984f5d 000000000098492a 00007fd860327e6e 00007fd860323f4c 00007fd8604200eb 00007fd86041f5ba 00007fd86041ce9e 00007fd86bd0886c 00007fd86bd2d8cd 00007fd86bd38f51 00007fd86d7e4439 00007fd8703a4757
1214 20:52:27.203070 (+ 19898us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.206252 (+ 3182us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.245094 (+ 38842us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15842176,"tablet-delete.queue_time_us":3232351}
I20251214 20:52:27.249109 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_78f54ddbc4d7430fa694c4ce272be743" } modify_external_catalogs: true
I20251214 20:52:27.249372 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 329) took 3327 ms. Trace:
I20251214 20:52:27.249460 1856 rpcz_store.cc:276] 1214 20:52:23.922175 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:23.922600 (+ 425us) service_pool.cc:225] Handling call
1214 20:52:27.249182 (+3326582us) ts_tablet_manager.cc:1151] Deleting tablet 34df3a34b15b4fe19bd12f785b07a033
1214 20:52:27.249196 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.249348 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3326424}
I20251214 20:52:27.249781 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 338) took 3228 ms. Trace:
I20251214 20:52:27.249858 1856 rpcz_store.cc:276] 1214 20:52:24.021346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.021520 (+ 174us) service_pool.cc:225] Handling call
1214 20:52:27.249685 (+3228165us) ts_tablet_manager.cc:1151] Deleting tablet 3bed2c326906486e99300e05033c9c0f
1214 20:52:27.249693 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.249765 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3227915}
I20251214 20:52:27.250164 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 339) took 3225 ms. Trace:
I20251214 20:52:27.250131 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 (table default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]) successfully deleted
I20251214 20:52:27.250245 1856 rpcz_store.cc:276] 1214 20:52:24.024387 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.024438 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:27.250069 (+3225631us) ts_tablet_manager.cc:1151] Deleting tablet 1d7aa508b11944aa8e268741c1998a9e
1214 20:52:27.250076 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.250147 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3225417}
W20251214 20:52:27.250459 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 34df3a34b15b4fe19bd12f785b07a033 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34df3a34b15b4fe19bd12f785b07a033 already in progress: creating tablet
I20251214 20:52:27.250550 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 347) took 3080 ms. Trace:
I20251214 20:52:27.250607 1856 rpcz_store.cc:276] 1214 20:52:24.170538 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.170618 (+ 80us) service_pool.cc:225] Handling call
1214 20:52:27.250457 (+3079839us) ts_tablet_manager.cc:1151] Deleting tablet feb19ea389af4ea6bb158a0e8aaeb094
1214 20:52:27.250466 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.250534 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3079467}
W20251214 20:52:27.250685 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3bed2c326906486e99300e05033c9c0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bed2c326906486e99300e05033c9c0f already in progress: creating tablet
I20251214 20:52:27.250850 1856 tablet_replica.cc:333] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:27.250891 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 1d7aa508b11944aa8e268741c1998a9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d7aa508b11944aa8e268741c1998a9e already in progress: creating tablet
I20251214 20:52:27.251055 1856 raft_consensus.cc:2243] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.245607 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e621f29c01448399d22615284fccd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 202b1d53cab644c98838b0ee81f29f22) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.251292 1856 raft_consensus.cc:2272] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.251926 1856 ts_tablet_manager.cc:1916] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:27.251101 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet feb19ea389af4ea6bb158a0e8aaeb094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet feb19ea389af4ea6bb158a0e8aaeb094 already in progress: creating tablet
W20251214 20:52:27.252822 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 3bd5c18696f54ea190c1ef84b93c4f63 (table default.Table_a889ffcf345c4209b084b35515555d67 [id=c3c58892ff8d4055a33380fd52feb3b1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3bd5c18696f54ea190c1ef84b93c4f63 consensus state has no leader
I20251214 20:52:27.255740 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_8e83dc5fa8e547f6bfdc4ddba95fcb78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.257021 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e83dc5fa8e547f6bfdc4ddba95fcb78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:27.230748 851 consensus_peers.cc:597] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16816fa6b8c34207bb929147910b93cb. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:27.260099 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e621f29c01448399d22615284fccd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 202b1d53cab644c98838b0ee81f29f22) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.212232 2287 raft_consensus.cc:515] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:27.270694 2287 leader_election.cc:290] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:27.270807 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bdcecba5534294aca1eb419f18e9d2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:27.271037 1052 raft_consensus.cc:3060] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.273633 1052 raft_consensus.cc:2468] T 42bdcecba5534294aca1eb419f18e9d2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:27.274564 851 leader_election.cc:304] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:27.275058 2287 raft_consensus.cc:2804] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:27.275429 2523 raft_consensus.cc:493] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.275449 2287 raft_consensus.cc:697] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:27.275735 2523 raft_consensus.cc:515] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.275868 2287 consensus_queue.cc:237] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:27.277433 713 leader_election.cc:343] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 012b81223d0f4ae9b742014e329f11a9
W20251214 20:52:27.278424 712 leader_election.cc:343] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 012b81223d0f4ae9b742014e329f11a9
I20251214 20:52:27.278533 712 leader_election.cc:304] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:27.278780 2468 raft_consensus.cc:2749] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.278870 2523 leader_election.cc:290] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:27.282915 1202 tablet_bootstrap.cc:654] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.283051 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bdcecba5534294aca1eb419f18e9d2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:27.284416 775 raft_consensus.cc:3060] T 4408013cf49e46d0b18d24b48483ea49 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.291425 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bdcecba5534294aca1eb419f18e9d2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:27.292266 848 leader_election.cc:343] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:27.295255 848 leader_election.cc:343] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:27.296267 848 consensus_peers.cc:597] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:27.296842 1982 ts_tablet_manager.cc:1929] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:27.297889 1856 ts_tablet_manager.cc:1929] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.298642 1982 log.cc:1199] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/7d4626d02ca548f3bf5a1a9142600bff
I20251214 20:52:27.299767 1982 ts_tablet_manager.cc:1950] T 7d4626d02ca548f3bf5a1a9142600bff P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:27.203058 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 301) took 3112 ms. Trace:
I20251214 20:52:27.301625 1202 tablet_bootstrap.cc:492] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:27.301941 1977 rpcz_store.cc:276] 1214 20:52:24.090801 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.090846 (+ 45us) service_pool.cc:225] Handling call
1214 20:52:27.202077 (+3111231us) ts_tablet_manager.cc:1151] Deleting tablet fe24e4f4e74040249068505058c15622
1214 20:52:27.202093 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.202943 (+ 850us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3110972}
W20251214 20:52:27.302824 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet fe24e4f4e74040249068505058c15622 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24e4f4e74040249068505058c15622 already in progress: creating tablet
I20251214 20:52:27.305044 1202 ts_tablet_manager.cc:1403] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.000s
I20251214 20:52:27.303519 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 332) took 2570 ms. Trace:
I20251214 20:52:27.305748 1982 rpcz_store.cc:276] 1214 20:52:24.733008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.733076 (+ 68us) service_pool.cc:225] Handling call
1214 20:52:27.207369 (+2474293us) ts_tablet_manager.cc:1151] Deleting tablet 7d4626d02ca548f3bf5a1a9142600bff
1214 20:52:27.207380 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.258769 (+ 51389us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.296681 (+ 37912us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.303498 (+ 6817us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2474035}
I20251214 20:52:27.307332 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 333) took 2353 ms. Trace:
I20251214 20:52:27.307421 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]) successfully deleted
I20251214 20:52:27.308259 1856 log.cc:1199] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ac64f4fcb6a943668584658df19f950c
I20251214 20:52:27.200469 1203 raft_consensus.cc:359] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:27.309427 1982 rpcz_store.cc:276] 1214 20:52:24.953820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.954697 (+ 877us) service_pool.cc:225] Handling call
1214 20:52:27.307148 (+2352451us) ts_tablet_manager.cc:1151] Deleting tablet 57f2314d33fa474ca36dafe0a0eae5ae
1214 20:52:27.307162 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.307300 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352108}
I20251214 20:52:27.309655 1856 ts_tablet_manager.cc:1950] T ac64f4fcb6a943668584658df19f950c P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:27.310129 1977 tablet_replica.cc:333] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:27.310447 1977 raft_consensus.cc:2243] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:27.310746 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 348) took 3130 ms. Trace:
I20251214 20:52:27.310952 1977 raft_consensus.cc:2272] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.310840 1856 rpcz_store.cc:276] 1214 20:52:24.180445 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.180504 (+ 59us) service_pool.cc:225] Handling call
1214 20:52:27.250802 (+3070298us) ts_tablet_manager.cc:1151] Deleting tablet ac64f4fcb6a943668584658df19f950c
1214 20:52:27.250810 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.270496 (+ 19686us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.297745 (+ 27249us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.310726 (+ 12981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3067268}
I20251214 20:52:27.310144 1202 raft_consensus.cc:359] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.311309 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 350) took 2941 ms. Trace:
I20251214 20:52:27.311376 1856 rpcz_store.cc:276] 1214 20:52:24.369416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.369686 (+ 270us) service_pool.cc:225] Handling call
1214 20:52:27.311156 (+2941470us) ts_tablet_manager.cc:1151] Deleting tablet 43fe3bb1ed3941ce9078376f8f478408
1214 20:52:27.311177 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.311294 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2926546}
I20251214 20:52:27.311501 1202 raft_consensus.cc:385] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.309675 1203 raft_consensus.cc:385] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.312199 1203 raft_consensus.cc:740] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.312038 1977 ts_tablet_manager.cc:1916] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.312578 1203 consensus_queue.cc:260] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:27.312922 1050 raft_consensus.cc:3060] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.315340 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ac64f4fcb6a943668584658df19f950c (table default.Table_5275181d46dc4a85b27fc0c45fa587fe [id=ab9838348eb24201bcb2866b16994965]) successfully deleted
I20251214 20:52:27.314524 1202 raft_consensus.cc:740] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
W20251214 20:52:27.315753 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 43fe3bb1ed3941ce9078376f8f478408 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43fe3bb1ed3941ce9078376f8f478408 already in progress: creating tablet
I20251214 20:52:27.316008 1050 raft_consensus.cc:2468] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:27.321370 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 351) took 2942 ms. Trace:
I20251214 20:52:27.321964 1856 rpcz_store.cc:276] 1214 20:52:24.369527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.369573 (+ 46us) service_pool.cc:225] Handling call
1214 20:52:27.311598 (+2942025us) ts_tablet_manager.cc:1151] Deleting tablet 054cfd587d19409fbdf7f4848bcaf3f9
1214 20:52:27.311607 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.311677 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2926882}
I20251214 20:52:27.323113 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 356) took 2907 ms. Trace:
W20251214 20:52:27.323834 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 054cfd587d19409fbdf7f4848bcaf3f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 054cfd587d19409fbdf7f4848bcaf3f9 already in progress: creating tablet
I20251214 20:52:27.324214 1856 rpcz_store.cc:276] 1214 20:52:24.415382 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.468997 (+ 53615us) service_pool.cc:225] Handling call
1214 20:52:27.322929 (+2853932us) ts_tablet_manager.cc:1151] Deleting tablet 727e7b61ae3942098712720d344c5310
1214 20:52:27.322946 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.323094 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2739794}
I20251214 20:52:27.324625 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 376) took 2574 ms. Trace:
I20251214 20:52:27.324697 1856 rpcz_store.cc:276] 1214 20:52:24.750557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.750613 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:27.324501 (+2573888us) ts_tablet_manager.cc:1151] Deleting tablet 54e5fff1fc84419ab3d3131cb3a12d92
1214 20:52:27.324511 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.324608 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2573397}
I20251214 20:52:27.324958 1856 tablet_replica.cc:333] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:27.323724 1202 consensus_queue.cc:260] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:27.327828 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 727e7b61ae3942098712720d344c5310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 727e7b61ae3942098712720d344c5310 already in progress: creating tablet
W20251214 20:52:27.328172 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 54e5fff1fc84419ab3d3131cb3a12d92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54e5fff1fc84419ab3d3131cb3a12d92 already in progress: creating tablet
I20251214 20:52:27.328878 1202 ts_tablet_manager.cc:1434] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20251214 20:52:27.331773 1856 raft_consensus.cc:2243] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:27.332765 1856 raft_consensus.cc:2272] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.333472 1202 tablet_bootstrap.cc:492] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:27.334200 1856 ts_tablet_manager.cc:1916] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.337282 1203 ts_tablet_manager.cc:1434] T 6fef1d6add764fd28a8f4995b337ea71 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.146s user 0.003s sys 0.000s
I20251214 20:52:27.338760 1203 tablet_bootstrap.cc:492] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:27.341056 1203 tablet_bootstrap.cc:654] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.345911 2327 raft_consensus.cc:493] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.346084 2327 raft_consensus.cc:515] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.346853 2327 leader_election.cc:290] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:27.348462 851 leader_election.cc:343] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 4d6e7306254044449f1127fb963ca5f0
W20251214 20:52:27.349867 848 leader_election.cc:343] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Not found: Tablet not found: 4d6e7306254044449f1127fb963ca5f0
I20251214 20:52:27.349979 848 leader_election.cc:304] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:27.350466 2327 raft_consensus.cc:2749] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.351787 1856 ts_tablet_manager.cc:1929] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:27.359948 1856 log.cc:1199] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/7d4626d02ca548f3bf5a1a9142600bff
I20251214 20:52:27.360524 1856 ts_tablet_manager.cc:1950] T 7d4626d02ca548f3bf5a1a9142600bff P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:27.362732 1203 tablet_bootstrap.cc:492] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:27.363875 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 377) took 2610 ms. Trace:
I20251214 20:52:27.365244 2364 raft_consensus.cc:493] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.370324 2364 raft_consensus.cc:515] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.371225 2364 leader_election.cc:290] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:27.372516 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "658e4fe0d188483b9ceb9eb3b3f20835" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:27.372797 915 raft_consensus.cc:2468] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:27.373298 991 leader_election.cc:304] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:27.373690 2364 raft_consensus.cc:2804] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:27.373764 2364 raft_consensus.cc:493] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:27.373823 2364 raft_consensus.cc:3060] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.376670 2364 raft_consensus.cc:515] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.372583 1856 rpcz_store.cc:276] 1214 20:52:24.756203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.756290 (+ 87us) service_pool.cc:225] Handling call
1214 20:52:27.324902 (+2568612us) ts_tablet_manager.cc:1151] Deleting tablet 7d4626d02ca548f3bf5a1a9142600bff
1214 20:52:27.324910 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.337414 (+ 12504us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.351658 (+ 14244us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.363850 (+ 12192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2568337}
I20251214 20:52:27.378502 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 378) took 2421 ms. Trace:
I20251214 20:52:27.378693 1856 rpcz_store.cc:276] 1214 20:52:24.956643 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.956692 (+ 49us) service_pool.cc:225] Handling call
1214 20:52:27.378336 (+2421644us) ts_tablet_manager.cc:1151] Deleting tablet 57f2314d33fa474ca36dafe0a0eae5ae
1214 20:52:27.378349 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.378482 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2381347}
I20251214 20:52:27.379215 1856 tablet_replica.cc:333] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:27.379422 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "658e4fe0d188483b9ceb9eb3b3f20835" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:27.379628 915 raft_consensus.cc:3060] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.378971 2364 leader_election.cc:290] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:27.379525 1856 raft_consensus.cc:2243] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.380688 1856 raft_consensus.cc:2272] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.381366 1856 ts_tablet_manager.cc:1916] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.382259 915 raft_consensus.cc:2468] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:27.383239 991 leader_election.cc:304] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:27.342835 2468 raft_consensus.cc:493] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.383589 2364 raft_consensus.cc:2804] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:27.383687 2364 raft_consensus.cc:697] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:27.383702 2468 raft_consensus.cc:515] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.383929 2364 consensus_queue.cc:237] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.384416 2468 leader_election.cc:290] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:27.384591 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]) successfully deleted
W20251214 20:52:27.384882 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 57f2314d33fa474ca36dafe0a0eae5ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57f2314d33fa474ca36dafe0a0eae5ae already in progress: creating tablet
I20251214 20:52:27.385486 1982 tablet_replica.cc:333] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:27.345222 1206 tablet_bootstrap.cc:492] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:27.385715 1982 raft_consensus.cc:2243] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.385797 1206 ts_tablet_manager.cc:1403] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.163s user 0.009s sys 0.013s
I20251214 20:52:27.385893 1982 raft_consensus.cc:2272] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.386636 1206 raft_consensus.cc:359] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.386788 1206 raft_consensus.cc:385] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.386831 1206 raft_consensus.cc:740] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.387005 1206 consensus_queue.cc:260] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.387584 1206 ts_tablet_manager.cc:1434] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:27.387902 1206 tablet_bootstrap.cc:492] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:27.389508 1206 tablet_bootstrap.cc:654] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.391521 1203 ts_tablet_manager.cc:1403] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
W20251214 20:52:27.391654 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 57f2314d33fa474ca36dafe0a0eae5ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57f2314d33fa474ca36dafe0a0eae5ae already in progress: creating tablet
I20251214 20:52:27.373553 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "658e4fe0d188483b9ceb9eb3b3f20835" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:27.392475 884 tablet_service.cc:1552] Processing DeleteTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.392715 883 tablet_service.cc:1552] Processing DeleteTablet for tablet ec85857f047e44d78835305ddcb70768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.393107 1203 raft_consensus.cc:359] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.393268 1203 raft_consensus.cc:385] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.393319 1203 raft_consensus.cc:740] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.379096 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "658e4fe0d188483b9ceb9eb3b3f20835" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:27.383203 2523 raft_consensus.cc:493] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.394701 2523 raft_consensus.cc:515] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.395437 2523 leader_election.cc:290] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:27.395979 1203 consensus_queue.cc:260] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.399259 2550 tablet.cc:1722] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:27.399641 2327 raft_consensus.cc:493] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.399816 2327 raft_consensus.cc:515] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:27.400058 712 leader_election.cc:343] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 7dd657f3baff40bb8cb7f4648367f7a1
I20251214 20:52:27.400473 2327 leader_election.cc:290] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:27.401388 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec85857f047e44d78835305ddcb70768" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:27.401846 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec85857f047e44d78835305ddcb70768" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:27.402089 1052 raft_consensus.cc:2468] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:27.402647 848 leader_election.cc:343] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.402882 2468 raft_consensus.cc:493] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.403013 2468 raft_consensus.cc:515] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.403602 2468 leader_election.cc:290] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:27.404232 712 leader_election.cc:343] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: b2e83bca8ed642b6a1b705ef70fde27f
I20251214 20:52:27.404671 851 leader_election.cc:304] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:27.404773 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "190afbb866854dd7b5393e04443752b7" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:27.405007 2287 raft_consensus.cc:2804] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:27.405082 915 raft_consensus.cc:2393] T 190afbb866854dd7b5393e04443752b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:27.405580 2287 raft_consensus.cc:493] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:27.405880 2287 raft_consensus.cc:3060] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.407640 1856 ts_tablet_manager.cc:1929] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251214 20:52:27.408161 713 leader_election.cc:343] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 7dd657f3baff40bb8cb7f4648367f7a1
I20251214 20:52:27.408294 713 leader_election.cc:304] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:27.408555 2468 raft_consensus.cc:2749] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.409662 2327 consensus_queue.cc:1048] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:27.414629 1982 ts_tablet_manager.cc:1916] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:27.428936 848 consensus_peers.cc:597] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:27.429419 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 4aa1a44452074ec28a1bd9c48a5f47cb
I20251214 20:52:27.430445 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 1e31615d1277412eb8bd80451f2c4366
I20251214 20:52:27.431558 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet cea9423c40044e4cabb8bbd755a46688
I20251214 20:52:27.431770 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_f86bf75508ca43ada4372af83098ae35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:27.432202 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_fc1d7cb6f8544f6285ec8792deb1da74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.432504 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f86bf75508ca43ada4372af83098ae35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:27.432627 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc1d7cb6f8544f6285ec8792deb1da74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.433780 452 master_service.cc:954] ReplaceTablet: received request to replace tablet 1eff8a23d48c43e2bdbfb34f4bab6edf from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:27.435176 448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_1ba7876ea93447038c0de7f37fb52825" } modify_external_catalogs: true
I20251214 20:52:27.435349 448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_1ba7876ea93447038c0de7f37fb52825" } modify_external_catalogs: true
I20251214 20:52:27.435477 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_e74a0c96be654f4c808071da1ae734a0" } new_table_name: "default.Table_9698b9206b934e4db2decdf9c0180838" modify_external_catalogs: true
I20251214 20:52:27.436546 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa1a44452074ec28a1bd9c48a5f47cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.436764 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e31615d1277412eb8bd80451f2c4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.436906 749 tablet_service.cc:1552] Processing DeleteTablet for tablet cea9423c40044e4cabb8bbd755a46688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.408990 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2e83bca8ed642b6a1b705ef70fde27f" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:27.437296 1052 raft_consensus.cc:2468] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:27.437597 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_53e58d3cdbfe4ab2945a18979051d7a2" } new_table_name: "default.Table_15c1781afe3449ef87ef3c655ccca0df" modify_external_catalogs: true
I20251214 20:52:27.437716 1856 log.cc:1199] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ebb26a5ae7224e9ca8a513c7b5013618
I20251214 20:52:27.438531 1856 ts_tablet_manager.cc:1950] T ebb26a5ae7224e9ca8a513c7b5013618 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:27.438954 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 83869.5us
I20251214 20:52:27.439025 880 tablet_service.cc:1552] Processing DeleteTablet for tablet cea9423c40044e4cabb8bbd755a46688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.439221 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa1a44452074ec28a1bd9c48a5f47cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.439160 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_3377d29944b3480a88fd7c603ccd0a02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:27.439453 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e31615d1277412eb8bd80451f2c4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:27.439692 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3377d29944b3480a88fd7c603ccd0a02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.441149 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9a53d238203b418eabb29093faef194c" } new_table_name: "default.Table_1052ac11eee9483aa13db090acff5b7a" modify_external_catalogs: true
I20251214 20:52:27.442436 713 leader_election.cc:304] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:27.442745 2468 raft_consensus.cc:2804] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:27.442816 2468 raft_consensus.cc:493] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:27.442873 2468 raft_consensus.cc:3060] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.445516 452 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_913ff42ae53c475ab302d4e056e7eb48" } modify_external_catalogs: true
I20251214 20:52:27.445748 452 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_913ff42ae53c475ab302d4e056e7eb48" } modify_external_catalogs: true
I20251214 20:52:27.445716 2468 raft_consensus.cc:515] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.447540 1202 tablet_bootstrap.cc:654] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.448314 2468 leader_election.cc:290] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:27.450105 712 leader_election.cc:343] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: b2e83bca8ed642b6a1b705ef70fde27f
I20251214 20:52:27.451143 453 master_service.cc:954] ReplaceTablet: received request to replace tablet f84e9dfc23d34eb0a379e8ac5a442e59 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:27.451786 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_e4ed29947e6544b9a23370d584496d4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.452430 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4ed29947e6544b9a23370d584496d4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.454236 1982 ts_tablet_manager.cc:1929] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251214 20:52:27.429752 989 leader_election.cc:343] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.409041 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "190afbb866854dd7b5393e04443752b7" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:27.410422 1203 ts_tablet_manager.cc:1434] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20251214 20:52:27.441993 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e31615d1277412eb8bd80451f2c4366 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.442116 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet cea9423c40044e4cabb8bbd755a46688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.442203 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa1a44452074ec28a1bd9c48a5f47cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.449247 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2e83bca8ed642b6a1b705ef70fde27f" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:27.455735 1052 raft_consensus.cc:3060] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.455868 1203 tablet_bootstrap.cc:492] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:27.456311 2287 raft_consensus.cc:515] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.457451 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec85857f047e44d78835305ddcb70768" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:27.457612 1203 tablet_bootstrap.cc:654] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.457635 1050 raft_consensus.cc:3060] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.458572 1052 raft_consensus.cc:2468] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
W20251214 20:52:27.458853 989 leader_election.cc:343] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.460255 1050 raft_consensus.cc:2468] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:27.461146 2287 leader_election.cc:290] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:27.461342 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 379) took 2458 ms. Trace:
I20251214 20:52:27.462651 851 leader_election.cc:304] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:27.463222 1977 ts_tablet_manager.cc:1929] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.463634 1977 log.cc:1199] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/16816fa6b8c34207bb929147910b93cb
I20251214 20:52:27.463572 1202 tablet_bootstrap.cc:492] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:27.464818 1202 ts_tablet_manager.cc:1403] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.133s user 0.006s sys 0.000s
I20251214 20:52:27.466511 1977 ts_tablet_manager.cc:1950] T 16816fa6b8c34207bb929147910b93cb P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:27.466581 2569 tablet.cc:1722] T 8fcdd6bafa5c4aa6b59668cde523d010 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:27.462610 1856 rpcz_store.cc:276] 1214 20:52:25.002495 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.002554 (+ 59us) service_pool.cc:225] Handling call
1214 20:52:27.379116 (+2376562us) ts_tablet_manager.cc:1151] Deleting tablet ebb26a5ae7224e9ca8a513c7b5013618
1214 20:52:27.379168 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.388423 (+ 9255us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.407438 (+ 19015us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.461324 (+ 53886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2376353}
I20251214 20:52:27.468397 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ebb26a5ae7224e9ca8a513c7b5013618 (table default.Table_4907bb640d114586b98e6fa0e514cb5a [id=2a6d9d4658f24eb5b39adea101a0891c]) successfully deleted
I20251214 20:52:27.468832 1203 tablet_bootstrap.cc:492] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:27.468981 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 382) took 2111 ms. Trace:
I20251214 20:52:27.469115 1203 ts_tablet_manager.cc:1403] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20251214 20:52:27.469157 1856 rpcz_store.cc:276] 1214 20:52:25.357182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.357239 (+ 57us) service_pool.cc:225] Handling call
1214 20:52:27.468824 (+2111585us) ts_tablet_manager.cc:1151] Deleting tablet 248e3666fd13434e97378eb5a98dfadd
1214 20:52:27.468836 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.468961 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2111393}
W20251214 20:52:27.469732 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 248e3666fd13434e97378eb5a98dfadd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 248e3666fd13434e97378eb5a98dfadd already in progress: creating tablet
I20251214 20:52:27.470144 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 383) took 2099 ms. Trace:
I20251214 20:52:27.470307 1856 rpcz_store.cc:276] 1214 20:52:25.371114 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.371196 (+ 82us) service_pool.cc:225] Handling call
1214 20:52:27.470040 (+2098844us) ts_tablet_manager.cc:1151] Deleting tablet e6ebfc373fbf41b1920c271a15302484
1214 20:52:27.470049 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.470128 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2098679}
I20251214 20:52:27.470729 2564 raft_consensus.cc:2804] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:27.470997 1856 tablet_replica.cc:333] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:27.471163 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet e6ebfc373fbf41b1920c271a15302484 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6ebfc373fbf41b1920c271a15302484 already in progress: creating tablet
I20251214 20:52:27.471366 2566 tablet.cc:1722] T 8fcdd6bafa5c4aa6b59668cde523d010 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:27.470350 1203 raft_consensus.cc:359] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.471560 1203 raft_consensus.cc:385] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.471683 1203 raft_consensus.cc:740] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.471383 1856 raft_consensus.cc:2243] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.472007 1856 raft_consensus.cc:2272] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.472558 1856 ts_tablet_manager.cc:1916] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:27.473114 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76472ce9005b4bd2bbc51278816e8ae0 consensus state has no leader
I20251214 20:52:27.473529 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_70cb422b9a3d439cba4cddcdd9b31c73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.474119 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70cb422b9a3d439cba4cddcdd9b31c73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.474587 2327 consensus_queue.cc:1048] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:27.472011 1203 consensus_queue.cc:260] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:27.475374 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 639cf3b7654a4ae896c6edd8b2ff1844 consensus state has no leader
I20251214 20:52:27.475687 1203 ts_tablet_manager.cc:1434] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:27.475955 1203 tablet_bootstrap.cc:492] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:27.476033 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc8d725e6ac440899fc1cde2f7b546aa consensus state has no leader
I20251214 20:52:27.477484 1203 tablet_bootstrap.cc:654] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.481431 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 300) took 3381 ms. Trace:
I20251214 20:52:27.483841 1977 rpcz_store.cc:276] 1214 20:52:24.090685 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.090979 (+ 294us) service_pool.cc:225] Handling call
1214 20:52:27.310047 (+3219068us) ts_tablet_manager.cc:1151] Deleting tablet 16816fa6b8c34207bb929147910b93cb
1214 20:52:27.310062 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.345612 (+ 35550us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.462730 (+117118us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.472425 (+ 9695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3218920}
I20251214 20:52:27.455253 713 leader_election.cc:304] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:27.484586 1202 raft_consensus.cc:359] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.485783 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 16816fa6b8c34207bb929147910b93cb (table default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]) successfully deleted
I20251214 20:52:27.485852 713 leader_election.cc:304] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters: a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:27.486012 1202 raft_consensus.cc:385] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.486464 2523 raft_consensus.cc:3060] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.486385 1202 raft_consensus.cc:740] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.486964 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8ccdf53e3ef149e1a70a20de6caedafa" } new_table_name: "default.Table_331c1dbc1c1f448e9f09c5fc935f9894" modify_external_catalogs: true
I20251214 20:52:27.487474 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 302) took 3396 ms. Trace:
I20251214 20:52:27.488909 2523 raft_consensus.cc:2749] T 190afbb866854dd7b5393e04443752b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.489099 2523 raft_consensus.cc:2804] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:27.489153 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_eaf150707b2442f1ae6f94ac564c6b87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:27.489574 2523 raft_consensus.cc:697] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:27.487852 1977 rpcz_store.cc:276] 1214 20:52:24.091228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.091309 (+ 81us) service_pool.cc:225] Handling call
1214 20:52:27.486346 (+3395037us) ts_tablet_manager.cc:1151] Deleting tablet 274185a4115b458e8e9a376bdbcc870b
1214 20:52:27.486358 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.487270 (+ 912us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3394852}
W20251214 20:52:27.489871 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaf150707b2442f1ae6f94ac564c6b87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.489851 2523 consensus_queue.cc:237] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:27.490478 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 274185a4115b458e8e9a376bdbcc870b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 274185a4115b458e8e9a376bdbcc870b already in progress: creating tablet
I20251214 20:52:27.488003 1202 consensus_queue.cc:260] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.461911 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ec85857f047e44d78835305ddcb70768" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:27.492388 848 leader_election.cc:343] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.492425 2574 tablet.cc:1722] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:27.493125 2550 tablet.cc:1722] T 8fcdd6bafa5c4aa6b59668cde523d010 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:27.493064 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 299) took 3403 ms. Trace:
I20251214 20:52:27.493520 1977 rpcz_store.cc:276] 1214 20:52:24.090573 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.091331 (+ 758us) service_pool.cc:225] Handling call
1214 20:52:27.492584 (+3401253us) ts_tablet_manager.cc:1151] Deleting tablet 8af99e0bd8774a678575847a9ec2e557
1214 20:52:27.492596 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.493045 (+ 449us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3400704}
W20251214 20:52:27.494705 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 8af99e0bd8774a678575847a9ec2e557 because the tablet was not found. No further retry: Not found: Tablet not found: 8af99e0bd8774a678575847a9ec2e557
I20251214 20:52:27.496021 1982 log.cc:1199] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ebb26a5ae7224e9ca8a513c7b5013618
I20251214 20:52:27.498278 1982 ts_tablet_manager.cc:1950] T ebb26a5ae7224e9ca8a513c7b5013618 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:27.499893 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 334) took 2497 ms. Trace:
I20251214 20:52:27.500005 1202 ts_tablet_manager.cc:1434] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
I20251214 20:52:27.501024 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 293) took 3415 ms. Trace:
I20251214 20:52:27.500240 1982 rpcz_store.cc:276] 1214 20:52:25.002524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.009429 (+ 6905us) service_pool.cc:225] Handling call
1214 20:52:27.385420 (+2375991us) ts_tablet_manager.cc:1151] Deleting tablet ebb26a5ae7224e9ca8a513c7b5013618
1214 20:52:27.385435 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.417651 (+ 32216us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.454117 (+ 36466us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.499877 (+ 45760us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1996032,"tablet-delete.queue_time_us":2375764}
I20251214 20:52:27.502285 1202 tablet_bootstrap.cc:492] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:27.501559 1977 rpcz_store.cc:276] 1214 20:52:24.084951 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.086882 (+ 1931us) service_pool.cc:225] Handling call
1214 20:52:27.499896 (+3413014us) ts_tablet_manager.cc:1151] Deleting tablet 1d7aa508b11944aa8e268741c1998a9e
1214 20:52:27.499908 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.500780 (+ 872us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3398533}
I20251214 20:52:27.438979 2549 tablet.cc:1722] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251214 20:52:27.503602 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 1d7aa508b11944aa8e268741c1998a9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d7aa508b11944aa8e268741c1998a9e already in progress: creating tablet
I20251214 20:52:27.505242 1977 tablet_replica.cc:333] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:27.503616 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ebb26a5ae7224e9ca8a513c7b5013618 (table default.Table_4907bb640d114586b98e6fa0e514cb5a [id=2a6d9d4658f24eb5b39adea101a0891c]) successfully deleted
I20251214 20:52:27.507925 1977 raft_consensus.cc:2243] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.511883 1977 raft_consensus.cc:2272] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.513402 1977 ts_tablet_manager.cc:1916] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.514541 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 335) took 2162 ms. Trace:
W20251214 20:52:27.515053 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67104e34df294ef5986171eca79cdc72 consensus state has no leader
I20251214 20:52:27.516651 1982 rpcz_store.cc:276] 1214 20:52:25.351883 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.357225 (+ 5342us) service_pool.cc:225] Handling call
1214 20:52:27.514376 (+2157151us) ts_tablet_manager.cc:1151] Deleting tablet 248e3666fd13434e97378eb5a98dfadd
1214 20:52:27.514390 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.514511 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2153840}
W20251214 20:52:27.518172 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 248e3666fd13434e97378eb5a98dfadd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 248e3666fd13434e97378eb5a98dfadd already in progress: creating tablet
I20251214 20:52:27.519620 446 master_service.cc:954] ReplaceTablet: received request to replace tablet 79e621f29c01448399d22615284fccd4 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:27.521729 2523 raft_consensus.cc:493] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.521966 2523 raft_consensus.cc:515] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.522032 2564 raft_consensus.cc:697] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:27.522360 2564 consensus_queue.cc:237] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.523234 2523 leader_election.cc:290] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:27.523614 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "242a77ae57174de48f4eaf43384ee0ba" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:27.524045 1052 raft_consensus.cc:2468] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:27.524947 713 leader_election.cc:304] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:27.525242 1202 tablet_bootstrap.cc:654] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.525585 2523 raft_consensus.cc:2804] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:27.525704 2523 raft_consensus.cc:493] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:27.525786 2523 raft_consensus.cc:3060] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.528788 2523 raft_consensus.cc:515] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.530263 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "242a77ae57174de48f4eaf43384ee0ba" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:27.530563 1052 raft_consensus.cc:3060] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.532178 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 336) took 2162 ms. Trace:
I20251214 20:52:27.532379 1982 rpcz_store.cc:276] 1214 20:52:25.370174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.371079 (+ 905us) service_pool.cc:225] Handling call
1214 20:52:27.518576 (+2147497us) ts_tablet_manager.cc:1151] Deleting tablet e6ebfc373fbf41b1920c271a15302484
1214 20:52:27.524323 (+ 5747us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.532154 (+ 7831us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2147258}
W20251214 20:52:27.533088 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet e6ebfc373fbf41b1920c271a15302484 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6ebfc373fbf41b1920c271a15302484 already in progress: creating tablet
I20251214 20:52:27.534335 1052 raft_consensus.cc:2468] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:27.535605 713 leader_election.cc:304] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75; no voters:
I20251214 20:52:27.535926 2468 raft_consensus.cc:2804] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:27.536064 2468 raft_consensus.cc:697] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:27.536353 2468 consensus_queue.cc:237] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.538244 1856 ts_tablet_manager.cc:1929] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:27.538472 1856 log.cc:1199] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/d1389716f5ce44f588c21a34c7a7d4c8
I20251214 20:52:27.538870 1856 ts_tablet_manager.cc:1950] T d1389716f5ce44f588c21a34c7a7d4c8 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:27.539168 1982 tablet_replica.cc:333] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:27.539438 1982 raft_consensus.cc:2243] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.539672 1982 raft_consensus.cc:2272] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.541375 2523 leader_election.cc:290] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:27.542671 712 leader_election.cc:343] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 242a77ae57174de48f4eaf43384ee0ba
I20251214 20:52:27.542894 1206 tablet_bootstrap.cc:492] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:27.543048 1206 ts_tablet_manager.cc:1403] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.155s user 0.003s sys 0.005s
W20251214 20:52:27.543236 712 leader_election.cc:343] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 242a77ae57174de48f4eaf43384ee0ba
I20251214 20:52:27.543989 1206 raft_consensus.cc:359] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.544277 1982 ts_tablet_manager.cc:1916] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.544284 1206 raft_consensus.cc:385] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.544615 1206 raft_consensus.cc:740] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.544903 1206 consensus_queue.cc:260] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.545528 1206 ts_tablet_manager.cc:1434] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:27.545847 1206 tablet_bootstrap.cc:492] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:27.556048 1206 tablet_bootstrap.cc:654] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.566461 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 385) took 2027 ms. Trace:
I20251214 20:52:27.565613 1203 tablet_bootstrap.cc:492] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:27.566787 1203 ts_tablet_manager.cc:1403] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.091s user 0.000s sys 0.004s
I20251214 20:52:27.567634 1203 raft_consensus.cc:359] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:27.567871 1203 raft_consensus.cc:385] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.567970 1203 raft_consensus.cc:740] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.568281 1203 consensus_queue.cc:260] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:27.568975 1203 ts_tablet_manager.cc:1434] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:27.569283 1203 tablet_bootstrap.cc:492] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:27.569139 450 catalog_manager.cc:5654] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "074399864fb2470fa9e2ee87c1fe2212" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:27.571913 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 1c5ee4114cb34540af2c5b332ebe82ab
I20251214 20:52:27.572458 447 catalog_manager.cc:6724] ReplaceTablet: tablet 1c5ee4114cb34540af2c5b332ebe82ab deleted and replaced by tablet 7e0847d58c0d4c5cbd1f9b91fd103763
I20251214 20:52:27.573783 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c5ee4114cb34540af2c5b332ebe82ab with delete_type TABLET_DATA_DELETED (replaced by tablet 7e0847d58c0d4c5cbd1f9b91fd103763) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.574750 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c5ee4114cb34540af2c5b332ebe82ab with delete_type TABLET_DATA_DELETED (replaced by tablet 7e0847d58c0d4c5cbd1f9b91fd103763) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.576609 743 tablet_service.cc:1505] Processing CreateTablet for tablet 96a2b444608e42d687eeba2dc0ebb757 (DEFAULT_TABLE table=default.Table_6eb267b64a6a485bb607876002c23ec4 [id=7c3d3542270e449ca954d068bece3491]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.564602 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:27.573050 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c5ee4114cb34540af2c5b332ebe82ab with delete_type TABLET_DATA_DELETED (replaced by tablet 7e0847d58c0d4c5cbd1f9b91fd103763) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.577040 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a2b444608e42d687eeba2dc0ebb757. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.576988 884 tablet_service.cc:1505] Processing CreateTablet for tablet 96a2b444608e42d687eeba2dc0ebb757 (DEFAULT_TABLE table=default.Table_6eb267b64a6a485bb607876002c23ec4 [id=7c3d3542270e449ca954d068bece3491]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.577421 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a2b444608e42d687eeba2dc0ebb757. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.577785 1856 rpcz_store.cc:276] 1214 20:52:25.538884 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.538942 (+ 58us) service_pool.cc:225] Handling call
1214 20:52:27.470925 (+1931983us) ts_tablet_manager.cc:1151] Deleting tablet d1389716f5ce44f588c21a34c7a7d4c8
1214 20:52:27.470935 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.515083 (+ 44148us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.538112 (+ 23029us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.566441 (+ 28329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1590336}
I20251214 20:52:27.578358 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]) successfully deleted
I20251214 20:52:27.578673 1856 tablet_replica.cc:333] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:27.578850 1856 raft_consensus.cc:2243] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:27.579193 1856 raft_consensus.cc:2272] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.579732 1856 ts_tablet_manager.cc:1916] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.584538 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 96a2b444608e42d687eeba2dc0ebb757 (DEFAULT_TABLE table=default.Table_6eb267b64a6a485bb607876002c23ec4 [id=7c3d3542270e449ca954d068bece3491]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.584961 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a2b444608e42d687eeba2dc0ebb757. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.584848 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5d1ff53eee3c4eb596c1b728e7d3bc1d" } new_table_name: "default.Table_41d8afc33560463fa806ba1661978de9" modify_external_catalogs: true
I20251214 20:52:27.586475 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_906242279a5b43858d131c378a5ea8da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.587020 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_906242279a5b43858d131c378a5ea8da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.594501 1203 tablet_bootstrap.cc:654] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.596873 1982 ts_tablet_manager.cc:1929] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:27.603099 1977 ts_tablet_manager.cc:1929] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.603410 1977 log.cc:1199] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/2d164747cb294bd4b126b4420231867d
I20251214 20:52:27.603879 1977 ts_tablet_manager.cc:1950] T 2d164747cb294bd4b126b4420231867d P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:27.604663 452 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 6d535d8f71b44c81bbba6937e804d900
I20251214 20:52:27.605006 2571 raft_consensus.cc:493] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.605134 2571 raft_consensus.cc:515] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.605791 2571 leader_election.cc:290] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:27.606244 448 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 9e1835374cfa48939d72476870accc3b
I20251214 20:52:27.606818 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c973e0111804b8b869068bdcc5869bf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:27.607041 1052 raft_consensus.cc:2468] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:27.607439 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_7694eee14c2346a9a71802bdf0897011"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.608146 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7694eee14c2346a9a71802bdf0897011 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.607951 455 catalog_manager.cc:5654] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:27.608417 455 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:27.608448 851 leader_election.cc:304] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:27.608774 455 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:27.609647 2572 raft_consensus.cc:2804] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:27.609741 2572 raft_consensus.cc:493] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:27.609833 2572 raft_consensus.cc:3060] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.610224 1982 log.cc:1199] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/d1389716f5ce44f588c21a34c7a7d4c8
I20251214 20:52:27.610358 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d535d8f71b44c81bbba6937e804d900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.610575 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e1835374cfa48939d72476870accc3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.610565 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e1835374cfa48939d72476870accc3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.610769 745 tablet_service.cc:1552] Processing DeleteTablet for tablet ebb26a5ae7224e9ca8a513c7b5013618 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.610881 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dd657f3baff40bb8cb7f4648367f7a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.610980 747 tablet_service.cc:1552] Processing DeleteTablet for tablet b2e83bca8ed642b6a1b705ef70fde27f with delete_type TABLET_DATA_DELETED (replaced by tablet f11834cabdad445b8e53cd237d146282) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.611855 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d535d8f71b44c81bbba6937e804d900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.612026 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d535d8f71b44c81bbba6937e804d900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.612052 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e1835374cfa48939d72476870accc3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.612709 2572 raft_consensus.cc:515] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.613473 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 242a77ae57174de48f4eaf43384ee0ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.614156 2564 consensus_queue.cc:1048] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:27.614849 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c973e0111804b8b869068bdcc5869bf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:27.615428 848 leader_election.cc:343] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.616813 883 tablet_service.cc:1505] Processing CreateTablet for tablet 8a977db125314550891491dcc53135cd (DEFAULT_TABLE table=default.Table_17954423e5674f0eb5a9b53bc2e75913 [id=c81e9d8bbee64045af7fce5cc8f64779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.617246 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a977db125314550891491dcc53135cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.617650 746 tablet_service.cc:1505] Processing CreateTablet for tablet 8a977db125314550891491dcc53135cd (DEFAULT_TABLE table=default.Table_17954423e5674f0eb5a9b53bc2e75913 [id=c81e9d8bbee64045af7fce5cc8f64779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.618060 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a977db125314550891491dcc53135cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:27.622138 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2807af0754914688bd7a090b1bc32eb1 consensus state has no leader
I20251214 20:52:27.622238 1856 ts_tablet_manager.cc:1929] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251214 20:52:27.622337 1856 log.cc:1199] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/a142157acbcd44adb6099b4428103ee5
I20251214 20:52:27.622658 1856 ts_tablet_manager.cc:1950] T a142157acbcd44adb6099b4428103ee5 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:27.623430 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 397) took 1727 ms. Trace:
I20251214 20:52:27.627772 2564 raft_consensus.cc:493] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.628064 2564 raft_consensus.cc:515] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.628654 1856 rpcz_store.cc:276] 1214 20:52:25.895834 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.895888 (+ 54us) service_pool.cc:225] Handling call
1214 20:52:27.578620 (+1682732us) ts_tablet_manager.cc:1151] Deleting tablet a142157acbcd44adb6099b4428103ee5
1214 20:52:27.578632 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.612068 (+ 33436us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.622120 (+ 10052us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.623397 (+ 1277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682520}
W20251214 20:52:27.629570 851 leader_election.cc:343] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: d1389716f5ce44f588c21a34c7a7d4c8
I20251214 20:52:27.629777 447 master_service.cc:954] ReplaceTablet: received request to replace tablet ed9790fbebda4f848291930bf1dd8174 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:27.629974 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet a142157acbcd44adb6099b4428103ee5 (table default.Table_6a7108611108447db74a6feddaeea2b5 [id=0a848dd5f6474712b2eba70add3743dc]) successfully deleted
I20251214 20:52:27.630173 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 399) took 1678 ms. Trace:
I20251214 20:52:27.630381 1856 rpcz_store.cc:276] 1214 20:52:25.951107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.951314 (+ 207us) service_pool.cc:225] Handling call
1214 20:52:27.629326 (+1678012us) ts_tablet_manager.cc:1151] Deleting tablet 2f97e5bad46343098418e8e34334b892
1214 20:52:27.629337 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.629459 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1677805}
I20251214 20:52:27.631091 2564 leader_election.cc:290] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:27.631263 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2f97e5bad46343098418e8e34334b892 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f97e5bad46343098418e8e34334b892 already in progress: creating tablet
I20251214 20:52:27.631704 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1389716f5ce44f588c21a34c7a7d4c8" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:27.631770 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 400) took 1680 ms. Trace:
W20251214 20:52:27.632217 848 leader_election.cc:343] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: Tablet not RUNNING: STOPPED
I20251214 20:52:27.632326 848 leader_election.cc:304] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:27.633154 2571 raft_consensus.cc:2749] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.633718 2572 leader_election.cc:290] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:27.633670 743 tablet_service.cc:1505] Processing CreateTablet for tablet c7248a2e8355425fa051a197160dbdbb (DEFAULT_TABLE table=default.Table_4bb4b496a62d4c89b7b4e783023dd5cd [id=144f1eb0434a44f49e52b76ba687525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.634132 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7248a2e8355425fa051a197160dbdbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.634063 880 tablet_service.cc:1505] Processing CreateTablet for tablet c7248a2e8355425fa051a197160dbdbb (DEFAULT_TABLE table=default.Table_4bb4b496a62d4c89b7b4e783023dd5cd [id=144f1eb0434a44f49e52b76ba687525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.634533 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7248a2e8355425fa051a197160dbdbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.596903 1206 tablet_bootstrap.cc:492] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:27.607844 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c973e0111804b8b869068bdcc5869bf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:27.638283 848 leader_election.cc:343] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.638736 1206 ts_tablet_manager.cc:1403] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.000s
I20251214 20:52:27.640825 1206 raft_consensus.cc:359] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.640996 1206 raft_consensus.cc:385] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.641040 1206 raft_consensus.cc:740] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.633559 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c973e0111804b8b869068bdcc5869bf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:27.641357 1050 raft_consensus.cc:3060] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.641254 1856 rpcz_store.cc:276] 1214 20:52:25.951231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.951321 (+ 90us) service_pool.cc:225] Handling call
1214 20:52:27.631652 (+1680331us) ts_tablet_manager.cc:1151] Deleting tablet c9488c3505f5443998531cf749722be9
1214 20:52:27.631663 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.631754 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1680019}
W20251214 20:52:27.641816 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c9488c3505f5443998531cf749722be9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9488c3505f5443998531cf749722be9 already in progress: creating tablet
I20251214 20:52:27.637202 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 8a977db125314550891491dcc53135cd (DEFAULT_TABLE table=default.Table_17954423e5674f0eb5a9b53bc2e75913 [id=c81e9d8bbee64045af7fce5cc8f64779]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.642364 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a977db125314550891491dcc53135cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.642549 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 398) took 1691 ms. Trace:
I20251214 20:52:27.641613 451 catalog_manager.cc:5654] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:27.642638 1856 rpcz_store.cc:276] 1214 20:52:25.950983 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.960880 (+ 9897us) service_pool.cc:225] Handling call
1214 20:52:27.642392 (+1681512us) ts_tablet_manager.cc:1151] Deleting tablet 1eff8a23d48c43e2bdbfb34f4bab6edf
1214 20:52:27.642418 (+ 26us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.642528 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1681268}
W20251214 20:52:27.643055 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1eff8a23d48c43e2bdbfb34f4bab6edf already in progress: creating tablet
I20251214 20:52:27.644083 451 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:27.644212 1050 raft_consensus.cc:2468] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:27.644522 451 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:27.645776 1206 consensus_queue.cc:260] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.646325 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet ed9790fbebda4f848291930bf1dd8174
I20251214 20:52:27.646746 447 catalog_manager.cc:6724] ReplaceTablet: tablet ed9790fbebda4f848291930bf1dd8174 deleted and replaced by tablet 76d16819e133434eb1c5a65fffb1f03f
I20251214 20:52:27.646910 1206 ts_tablet_manager.cc:1434] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251214 20:52:27.647548 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 401) took 1668 ms. Trace:
I20251214 20:52:27.647658 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet ed9790fbebda4f848291930bf1dd8174 with delete_type TABLET_DATA_DELETED (replaced by tablet 76d16819e133434eb1c5a65fffb1f03f) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.648146 1856 rpcz_store.cc:276] 1214 20:52:25.979359 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.046113 (+ 66754us) service_pool.cc:225] Handling call
1214 20:52:27.647388 (+1601275us) ts_tablet_manager.cc:1151] Deleting tablet 57a6232564194cbba12182f3a29c2ebb
1214 20:52:27.647400 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.647526 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600816}
I20251214 20:52:27.648976 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 403) took 1345 ms. Trace:
I20251214 20:52:27.649446 1206 tablet_bootstrap.cc:492] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:27.649813 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 57a6232564194cbba12182f3a29c2ebb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57a6232564194cbba12182f3a29c2ebb already in progress: creating tablet
I20251214 20:52:27.650777 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 303) took 3559 ms. Trace:
I20251214 20:52:27.650842 851 leader_election.cc:304] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:27.650887 1977 rpcz_store.cc:276] 1214 20:52:24.091299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.094794 (+ 3495us) service_pool.cc:225] Handling call
1214 20:52:27.504904 (+3410110us) ts_tablet_manager.cc:1151] Deleting tablet 2d164747cb294bd4b126b4420231867d
1214 20:52:27.504914 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.544606 (+ 39692us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.602946 (+ 58340us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.650747 (+ 47801us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2720,"tablet-delete.queue_time_us":3402066}
I20251214 20:52:27.651309 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 304) took 3547 ms. Trace:
I20251214 20:52:27.652460 448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9f0b2a3e0ef1496db7886462be5513c3" } modify_external_catalogs: true
I20251214 20:52:27.652702 448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9f0b2a3e0ef1496db7886462be5513c3" } modify_external_catalogs: true
I20251214 20:52:27.652280 879 tablet_service.cc:1552] Processing DeleteTablet for tablet ed9790fbebda4f848291930bf1dd8174 with delete_type TABLET_DATA_DELETED (replaced by tablet 76d16819e133434eb1c5a65fffb1f03f) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.653059 1977 rpcz_store.cc:276] 1214 20:52:24.103724 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.103771 (+ 47us) service_pool.cc:225] Handling call
1214 20:52:27.651197 (+3547426us) ts_tablet_manager.cc:1151] Deleting tablet a063083f54ad49d2a842fa080d44ed85
1214 20:52:27.651210 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.651291 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3547261}
I20251214 20:52:27.653924 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 308) took 3547 ms. Trace:
I20251214 20:52:27.653980 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1]) successfully deleted
W20251214 20:52:27.654328 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a063083f54ad49d2a842fa080d44ed85 because the tablet was not found. No further retry: Not found: Tablet not found: a063083f54ad49d2a842fa080d44ed85
I20251214 20:52:27.655140 1977 rpcz_store.cc:276] 1214 20:52:24.105949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.106004 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:27.653799 (+3547795us) ts_tablet_manager.cc:1151] Deleting tablet 3b47ca1a03944cf1847d3578856dafa7
1214 20:52:27.653810 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.653907 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3547614}
I20251214 20:52:27.655958 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 311) took 3549 ms. Trace:
W20251214 20:52:27.656168 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3b47ca1a03944cf1847d3578856dafa7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b47ca1a03944cf1847d3578856dafa7 already in progress: creating tablet
I20251214 20:52:27.656392 1977 rpcz_store.cc:276] 1214 20:52:24.106237 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.110616 (+ 4379us) service_pool.cc:225] Handling call
1214 20:52:27.655848 (+3545232us) ts_tablet_manager.cc:1151] Deleting tablet f342dafae1d04be3ba38f7c0e4f5d058
1214 20:52:27.655857 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.655942 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3542915}
I20251214 20:52:27.656371 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d8643dc61ad94d65864a5eade80548fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.657143 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8643dc61ad94d65864a5eade80548fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:27.657152 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet f342dafae1d04be3ba38f7c0e4f5d058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f342dafae1d04be3ba38f7c0e4f5d058 already in progress: creating tablet
I20251214 20:52:27.657488 1202 tablet_bootstrap.cc:492] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:27.657667 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 309) took 3551 ms. Trace:
I20251214 20:52:27.657795 1202 ts_tablet_manager.cc:1403] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.156s user 0.009s sys 0.000s
I20251214 20:52:27.658804 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_debb0f541d1b4ca3bc00c5e2c81c8428"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:27.658898 1202 raft_consensus.cc:359] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.659304 1202 raft_consensus.cc:385] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.659409 1202 raft_consensus.cc:740] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
W20251214 20:52:27.659446 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_debb0f541d1b4ca3bc00c5e2c81c8428 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.659766 1202 consensus_queue.cc:260] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.660131 2571 raft_consensus.cc:2804] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:27.660470 2571 raft_consensus.cc:697] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:27.660769 2571 consensus_queue.cc:237] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.661403 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_f6c14f9b22304e8aafbe1afab658ae71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.661885 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6c14f9b22304e8aafbe1afab658ae71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.663089 1202 ts_tablet_manager.cc:1434] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:27.664467 743 tablet_service.cc:1552] Processing DeleteTablet for tablet ed9790fbebda4f848291930bf1dd8174 with delete_type TABLET_DATA_DELETED (replaced by tablet 76d16819e133434eb1c5a65fffb1f03f) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.664734 1202 tablet_bootstrap.cc:492] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:27.665241 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3bc13ee5c12f414c8375152aafd7c0d0" } modify_external_catalogs: true
I20251214 20:52:27.665439 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3bc13ee5c12f414c8375152aafd7c0d0" } modify_external_catalogs: true
I20251214 20:52:27.667354 1202 tablet_bootstrap.cc:654] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.667418 452 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5000227c87004ed7a604c72840d805da" } modify_external_catalogs: true
I20251214 20:52:27.667570 452 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5000227c87004ed7a604c72840d805da" } modify_external_catalogs: true
I20251214 20:52:27.610584 1982 ts_tablet_manager.cc:1950] T d1389716f5ce44f588c21a34c7a7d4c8 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:27.671303 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 337) took 2133 ms. Trace:
I20251214 20:52:27.676337 1977 rpcz_store.cc:276] 1214 20:52:24.106052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.111178 (+ 5126us) service_pool.cc:225] Handling call
1214 20:52:27.657559 (+3546381us) ts_tablet_manager.cc:1151] Deleting tablet eb29fd8c85344ed8aa8d880e2ed240bc
1214 20:52:27.657568 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.657649 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3543269}
I20251214 20:52:27.680322 746 tablet_service.cc:1505] Processing CreateTablet for tablet aa5e66f9d8a8410da1f349788a0a8ad8 (DEFAULT_TABLE table=default.Table_5b1cf9007e0b4c3bae65a57ec193d8fa [id=cffdf6b3ab8b4d109c4552b0472fe6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.680974 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa5e66f9d8a8410da1f349788a0a8ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.681133 743 tablet_service.cc:1505] Processing CreateTablet for tablet 9bd89709abbc49809a2cdb6d0c29fdb8 (DEFAULT_TABLE table=default.Table_7e0acfc6d6e84dad84938bb81b1f375d [id=687eb544b4114bad8d2deb76388e2022]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.681384 448 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 52c05204502a45e18c2b862e21c6f361
I20251214 20:52:27.681782 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bd89709abbc49809a2cdb6d0c29fdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.682435 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 52c05204502a45e18c2b862e21c6f361 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.685372 747 tablet_service.cc:1505] Processing CreateTablet for tablet 018452d555e04d1082da2cec1812a1f5 (DEFAULT_TABLE table=default.Table_624b9386a78b44e994cde78539ccedc4 [id=7ea13a9516b340dabbb4916b419b8c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.685768 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 018452d555e04d1082da2cec1812a1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.686625 884 tablet_service.cc:1505] Processing CreateTablet for tablet aa5e66f9d8a8410da1f349788a0a8ad8 (DEFAULT_TABLE table=default.Table_5b1cf9007e0b4c3bae65a57ec193d8fa [id=cffdf6b3ab8b4d109c4552b0472fe6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.688737 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa5e66f9d8a8410da1f349788a0a8ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.688839 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 307) took 3583 ms. Trace:
I20251214 20:52:27.688936 1977 rpcz_store.cc:276] 1214 20:52:24.105838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.111185 (+ 5347us) service_pool.cc:225] Handling call
1214 20:52:27.688671 (+3577486us) ts_tablet_manager.cc:1151] Deleting tablet 58ac06f852fc4abea5cb213926002ca1
1214 20:52:27.688684 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.688816 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3568541}
W20251214 20:52:27.689399 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet eb29fd8c85344ed8aa8d880e2ed240bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb29fd8c85344ed8aa8d880e2ed240bc already in progress: creating tablet
I20251214 20:52:27.689668 879 tablet_service.cc:1505] Processing CreateTablet for tablet 018452d555e04d1082da2cec1812a1f5 (DEFAULT_TABLE table=default.Table_624b9386a78b44e994cde78539ccedc4 [id=7ea13a9516b340dabbb4916b419b8c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.690076 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 018452d555e04d1082da2cec1812a1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.686874 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 52c05204502a45e18c2b862e21c6f361 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.687856 880 tablet_service.cc:1505] Processing CreateTablet for tablet 9bd89709abbc49809a2cdb6d0c29fdb8 (DEFAULT_TABLE table=default.Table_7e0acfc6d6e84dad84938bb81b1f375d [id=687eb544b4114bad8d2deb76388e2022]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.694901 745 tablet_service.cc:1505] Processing CreateTablet for tablet aaf060ab2f71429a85dda19669e69677 (DEFAULT_TABLE table=default.Table_9321256005b04155b2f5a451ce5065fe [id=c5a7a8665d6647bdb19ed6ef27907fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.694901 748 tablet_service.cc:1505] Processing CreateTablet for tablet 2f75657d7bc44f6393c55ab119eb7dc2 (DEFAULT_TABLE table=default.Table_edb8b6390dba4303b887b41508f5f525 [id=e8a9bd4b63cf47babfa086577af02cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.695325 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaf060ab2f71429a85dda19669e69677. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.695488 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f75657d7bc44f6393c55ab119eb7dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.700507 887 tablet_service.cc:1505] Processing CreateTablet for tablet aaf060ab2f71429a85dda19669e69677 (DEFAULT_TABLE table=default.Table_9321256005b04155b2f5a451ce5065fe [id=c5a7a8665d6647bdb19ed6ef27907fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.702286 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaf060ab2f71429a85dda19669e69677. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.709398 883 tablet_service.cc:1505] Processing CreateTablet for tablet 2f75657d7bc44f6393c55ab119eb7dc2 (DEFAULT_TABLE table=default.Table_edb8b6390dba4303b887b41508f5f525 [id=e8a9bd4b63cf47babfa086577af02cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.709913 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f75657d7bc44f6393c55ab119eb7dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.713759 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bd89709abbc49809a2cdb6d0c29fdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.713951 1856 rpcz_store.cc:276] 1214 20:52:26.303923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.304083 (+ 160us) service_pool.cc:225] Handling call
1214 20:52:27.648857 (+1344774us) ts_tablet_manager.cc:1151] Deleting tablet e59b80ee919b4e1393fe317ed0e3a3d2
1214 20:52:27.648867 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.648960 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1344508}
I20251214 20:52:27.714604 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 404) took 1399 ms. Trace:
I20251214 20:52:27.715059 1856 rpcz_store.cc:276] 1214 20:52:26.314712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.319202 (+ 4490us) service_pool.cc:225] Handling call
1214 20:52:27.714456 (+1395254us) ts_tablet_manager.cc:1151] Deleting tablet d745583012b4465090d85c75470e0bc2
1214 20:52:27.714477 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.714586 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395022}
W20251214 20:52:27.715427 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet e59b80ee919b4e1393fe317ed0e3a3d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e59b80ee919b4e1393fe317ed0e3a3d2 already in progress: creating tablet
W20251214 20:52:27.715781 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet d745583012b4465090d85c75470e0bc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d745583012b4465090d85c75470e0bc2 already in progress: creating tablet
I20251214 20:52:27.715963 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 405) took 1349 ms. Trace:
I20251214 20:52:27.716084 1856 rpcz_store.cc:276] 1214 20:52:26.366615 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.368056 (+ 1441us) service_pool.cc:225] Handling call
1214 20:52:27.715854 (+1347798us) ts_tablet_manager.cc:1151] Deleting tablet e7b247b761c94c60ad3a468db371b30b
1214 20:52:27.715863 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.715947 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347587}
I20251214 20:52:27.716935 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 406) took 1219 ms. Trace:
I20251214 20:52:27.717126 1856 rpcz_store.cc:276] 1214 20:52:26.497190 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.497245 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:27.716840 (+1219595us) ts_tablet_manager.cc:1151] Deleting tablet f1eb4c3aa654480caaa69dbd5047365c
1214 20:52:27.716849 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.716921 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1219360}
W20251214 20:52:27.718120 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet e7b247b761c94c60ad3a468db371b30b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7b247b761c94c60ad3a468db371b30b already in progress: creating tablet
I20251214 20:52:27.672071 1982 rpcz_store.cc:276] 1214 20:52:25.538259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.538333 (+ 74us) service_pool.cc:225] Handling call
1214 20:52:27.532902 (+1994569us) ts_tablet_manager.cc:1151] Deleting tablet d1389716f5ce44f588c21a34c7a7d4c8
1214 20:52:27.538970 (+ 6068us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.551751 (+ 12781us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.596754 (+ 45003us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.671275 (+ 74521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1658821}
W20251214 20:52:27.718394 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f1eb4c3aa654480caaa69dbd5047365c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1eb4c3aa654480caaa69dbd5047365c already in progress: creating tablet
W20251214 20:52:27.718593 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 4f358df4d9414dcdb0d197f434e28118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f358df4d9414dcdb0d197f434e28118 already in progress: creating tablet
I20251214 20:52:27.718873 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]) successfully deleted
I20251214 20:52:27.719569 1856 tablet_replica.cc:333] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:27.719765 1856 raft_consensus.cc:2243] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:27.720440 1856 raft_consensus.cc:2272] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.720897 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_aedc10034f9947a2bf065dda745b96f3" } new_table_name: "default.Table_2ca54bf356e24a1ebca6f4a155b83044" modify_external_catalogs: true
I20251214 20:52:27.723465 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_89c0478b97144f35b1a0adeb64be2d9f" } new_table_name: "default.Table_a426957d72cc493393637b00a3b1e12d" modify_external_catalogs: true
I20251214 20:52:27.725217 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_41d8afc33560463fa806ba1661978de9" } new_table_name: "default.Table_3b90ca08fe374acfa5c4ff6bf90cabb4" modify_external_catalogs: true
I20251214 20:52:27.725749 1856 ts_tablet_manager.cc:1916] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.719668 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 350) took 1658 ms. Trace:
W20251214 20:52:27.727631 848 consensus_peers.cc:597] T ba7952290e2c4746802cffeb14971da0 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:27.726593 1982 rpcz_store.cc:276] 1214 20:52:26.061683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.061728 (+ 45us) service_pool.cc:225] Handling call
1214 20:52:27.718911 (+1657183us) ts_tablet_manager.cc:1151] Deleting tablet 1eff8a23d48c43e2bdbfb34f4bab6edf
1214 20:52:27.719073 (+ 162us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.719650 (+ 577us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1656979}
W20251214 20:52:27.728796 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1eff8a23d48c43e2bdbfb34f4bab6edf already in progress: creating tablet
I20251214 20:52:27.730198 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 351) took 1668 ms. Trace:
I20251214 20:52:27.731333 1206 tablet_bootstrap.cc:654] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.732000 1856 ts_tablet_manager.cc:1929] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.731046 1982 rpcz_store.cc:276] 1214 20:52:26.062047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.062090 (+ 43us) service_pool.cc:225] Handling call
1214 20:52:27.728921 (+1666831us) ts_tablet_manager.cc:1151] Deleting tablet 2f97e5bad46343098418e8e34334b892
1214 20:52:27.729212 (+ 291us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.730169 (+ 957us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1666602}
I20251214 20:52:27.732121 1856 log.cc:1199] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/2d164747cb294bd4b126b4420231867d
I20251214 20:52:27.732445 1856 ts_tablet_manager.cc:1950] T 2d164747cb294bd4b126b4420231867d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:27.732753 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2f97e5bad46343098418e8e34334b892 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f97e5bad46343098418e8e34334b892 already in progress: creating tablet
I20251214 20:52:27.733328 1856 tablet_replica.cc:333] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:27.733527 1856 raft_consensus.cc:2243] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:27.733589 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1]) successfully deleted
I20251214 20:52:27.733634 1982 tablet_replica.cc:333] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:27.733870 1202 tablet_bootstrap.cc:492] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:27.733877 1856 raft_consensus.cc:2272] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.734020 1202 ts_tablet_manager.cc:1403] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.000s
I20251214 20:52:27.676192 1028 tablet_service.cc:1505] Processing CreateTablet for tablet c7248a2e8355425fa051a197160dbdbb (DEFAULT_TABLE table=default.Table_4bb4b496a62d4c89b7b4e783023dd5cd [id=144f1eb0434a44f49e52b76ba687525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.680297 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 9bd89709abbc49809a2cdb6d0c29fdb8 (DEFAULT_TABLE table=default.Table_7e0acfc6d6e84dad84938bb81b1f375d [id=687eb544b4114bad8d2deb76388e2022]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.734752 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bd89709abbc49809a2cdb6d0c29fdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.735100 1202 raft_consensus.cc:359] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.735386 1202 raft_consensus.cc:385] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.735478 1202 raft_consensus.cc:740] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.735778 1202 consensus_queue.cc:260] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.682318 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 52c05204502a45e18c2b862e21c6f361 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.684783 1030 tablet_service.cc:1505] Processing CreateTablet for tablet aa5e66f9d8a8410da1f349788a0a8ad8 (DEFAULT_TABLE table=default.Table_5b1cf9007e0b4c3bae65a57ec193d8fa [id=cffdf6b3ab8b4d109c4552b0472fe6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.739552 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa5e66f9d8a8410da1f349788a0a8ad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.740301 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7248a2e8355425fa051a197160dbdbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.742471 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 297) took 3653 ms. Trace:
I20251214 20:52:27.742580 454 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:27.742651 1977 rpcz_store.cc:276] 1214 20:52:24.089105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.098849 (+ 9744us) service_pool.cc:225] Handling call
1214 20:52:27.742340 (+3643491us) ts_tablet_manager.cc:1151] Deleting tablet ca263a80702f4f64aa7c872c2be08207
1214 20:52:27.742353 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.742445 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3601927}
I20251214 20:52:27.742924 454 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
W20251214 20:52:27.743264 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ca263a80702f4f64aa7c872c2be08207 because the tablet was not found. No further retry: Not found: Tablet not found: ca263a80702f4f64aa7c872c2be08207
W20251214 20:52:27.744021 848 consensus_peers.cc:597] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac64f4fcb6a943668584658df19f950c. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:27.687903 2364 raft_consensus.cc:493] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.688899 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 018452d555e04d1082da2cec1812a1f5 (DEFAULT_TABLE table=default.Table_624b9386a78b44e994cde78539ccedc4 [id=7ea13a9516b340dabbb4916b419b8c03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.747020 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 018452d555e04d1082da2cec1812a1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.748515 1982 raft_consensus.cc:2243] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.749631 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 296) took 3660 ms. Trace:
I20251214 20:52:27.749723 1977 rpcz_store.cc:276] 1214 20:52:24.088993 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.089111 (+ 118us) service_pool.cc:225] Handling call
1214 20:52:27.749511 (+3660400us) ts_tablet_manager.cc:1151] Deleting tablet b315744c8cd043c5a8bdceec73a7e28b
1214 20:52:27.749520 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.749608 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3565548}
W20251214 20:52:27.750172 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet b315744c8cd043c5a8bdceec73a7e28b because the tablet was not found. No further retry: Not found: Tablet not found: b315744c8cd043c5a8bdceec73a7e28b
I20251214 20:52:27.750555 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 315) took 3553 ms. Trace:
I20251214 20:52:27.750623 1977 rpcz_store.cc:276] 1214 20:52:24.196818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.196893 (+ 75us) service_pool.cc:225] Handling call
1214 20:52:27.750433 (+3553540us) ts_tablet_manager.cc:1151] Deleting tablet feb19ea389af4ea6bb158a0e8aaeb094
1214 20:52:27.750443 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.750533 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3523961}
I20251214 20:52:27.750177 1982 raft_consensus.cc:2272] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:27.750991 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet feb19ea389af4ea6bb158a0e8aaeb094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet feb19ea389af4ea6bb158a0e8aaeb094 already in progress: creating tablet
I20251214 20:52:27.693410 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 2f75657d7bc44f6393c55ab119eb7dc2 (DEFAULT_TABLE table=default.Table_edb8b6390dba4303b887b41508f5f525 [id=e8a9bd4b63cf47babfa086577af02cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.751292 1977 tablet_replica.cc:333] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:27.751398 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f75657d7bc44f6393c55ab119eb7dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.751482 1977 raft_consensus.cc:2243] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:27.751828 1977 raft_consensus.cc:2272] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.752411 1977 ts_tablet_manager.cc:1916] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.755702 1203 tablet_bootstrap.cc:492] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:27.755826 1203 ts_tablet_manager.cc:1403] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.187s user 0.008s sys 0.002s
I20251214 20:52:27.756083 1202 ts_tablet_manager.cc:1434] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20251214 20:52:27.756493 1203 raft_consensus.cc:359] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.756645 1203 raft_consensus.cc:385] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.756692 1203 raft_consensus.cc:740] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.756873 1203 consensus_queue.cc:260] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.757305 1203 ts_tablet_manager.cc:1434] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251214 20:52:27.757580 1203 tablet_bootstrap.cc:492] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:27.758869 1203 tablet_bootstrap.cc:654] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.672923 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f2314d33fa474ca36dafe0a0eae5ae with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82068013e8471c9309bcf8e8604ece) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.759996 1022 tablet_service.cc:1505] Processing CreateTablet for tablet aaf060ab2f71429a85dda19669e69677 (DEFAULT_TABLE table=default.Table_9321256005b04155b2f5a451ce5065fe [id=c5a7a8665d6647bdb19ed6ef27907fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.762374 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aaf060ab2f71429a85dda19669e69677. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.763424 1202 tablet_bootstrap.cc:492] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:27.764061 2561 consensus_queue.cc:1048] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:27.765280 1206 tablet_bootstrap.cc:492] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:27.765247 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ae43058f9a39467b9919a9e1c335602f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:27.765743 1206 ts_tablet_manager.cc:1403] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.116s user 0.000s sys 0.004s
W20251214 20:52:27.765952 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae43058f9a39467b9919a9e1c335602f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.766201 2364 raft_consensus.cc:515] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.766866 1206 raft_consensus.cc:359] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.767201 1206 raft_consensus.cc:385] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.767303 1206 raft_consensus.cc:740] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.767490 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69f6108792fa4c2a9744e3248c76251f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:27.768021 989 leader_election.cc:343] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.767988 1206 consensus_queue.cc:260] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.768918 2364 leader_election.cc:290] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:27.771399 1856 ts_tablet_manager.cc:1916] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.767499 1982 ts_tablet_manager.cc:1916] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.774317 1206 ts_tablet_manager.cc:1434] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251214 20:52:27.768795 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69f6108792fa4c2a9744e3248c76251f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:27.775565 1206 tablet_bootstrap.cc:492] T ba7952290e2c4746802cffeb14971da0 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:27.776662 991 leader_election.cc:343] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.777249 991 leader_election.cc:304] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:27.777781 2364 raft_consensus.cc:2749] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.786478 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_59b03580c6ff4f28baeb68e68994d7dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.787345 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59b03580c6ff4f28baeb68e68994d7dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.790208 1203 tablet_bootstrap.cc:492] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:27.790423 1203 ts_tablet_manager.cc:1403] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.001s
I20251214 20:52:27.791234 1203 raft_consensus.cc:359] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.791481 1203 raft_consensus.cc:385] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.791606 1203 raft_consensus.cc:740] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.792023 1203 consensus_queue.cc:260] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.792747 1203 ts_tablet_manager.cc:1434] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:27.793073 1203 tablet_bootstrap.cc:492] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:27.794041 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 2f9def8e4fd449599820a63845397750
I20251214 20:52:27.795395 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f9def8e4fd449599820a63845397750 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.797670 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f9def8e4fd449599820a63845397750 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.797860 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f9def8e4fd449599820a63845397750 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.798691 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d2428076aa4e41dbb5b7cd6937e54a20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:27.799250 1203 tablet_bootstrap.cc:654] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:27.799336 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2428076aa4e41dbb5b7cd6937e54a20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.802467 1202 tablet_bootstrap.cc:654] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.806366 1856 ts_tablet_manager.cc:1929] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:27.809000 452 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet d171301941fa43908e05984c3b016007
I20251214 20:52:27.809520 1982 ts_tablet_manager.cc:1929] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251214 20:52:27.809697 1982 log.cc:1199] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/a142157acbcd44adb6099b4428103ee5
I20251214 20:52:27.810082 1982 ts_tablet_manager.cc:1950] T a142157acbcd44adb6099b4428103ee5 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:27.810423 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 349) took 1748 ms. Trace:
I20251214 20:52:27.810562 1982 rpcz_store.cc:276] 1214 20:52:26.061585 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.062284 (+ 699us) service_pool.cc:225] Handling call
1214 20:52:27.733293 (+1671009us) ts_tablet_manager.cc:1151] Deleting tablet a142157acbcd44adb6099b4428103ee5
1214 20:52:27.733307 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.787683 (+ 54376us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.809390 (+ 21707us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.810408 (+ 1018us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1670502}
I20251214 20:52:27.811200 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 352) took 1739 ms. Trace:
I20251214 20:52:27.811553 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet a142157acbcd44adb6099b4428103ee5 (table default.Table_6a7108611108447db74a6feddaeea2b5 [id=0a848dd5f6474712b2eba70add3743dc]) successfully deleted
I20251214 20:52:27.811985 1856 log.cc:1199] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ca57963bcfb54eec9fcbed82757d8387
I20251214 20:52:27.811810 743 tablet_service.cc:1552] Processing DeleteTablet for tablet d171301941fa43908e05984c3b016007 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.812031 748 tablet_service.cc:1505] Processing CreateTablet for tablet 205e951fef7445d79a900f256cff8074 (DEFAULT_TABLE table=default.Table_2541d743bd8945a2b0d984cdf911d6ba [id=825f8141b3ca4a1983f6548c66239504]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.813649 1982 rpcz_store.cc:276] 1214 20:52:26.072221 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.072314 (+ 93us) service_pool.cc:225] Handling call
1214 20:52:27.811055 (+1738741us) ts_tablet_manager.cc:1151] Deleting tablet c9488c3505f5443998531cf749722be9
1214 20:52:27.811065 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.811185 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1738351}
I20251214 20:52:27.812908 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet d171301941fa43908e05984c3b016007 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.813232 1036 tablet_service.cc:1505] Processing CreateTablet for tablet a6972d034dd54bd880dc09a3fb983d0c (DEFAULT_TABLE table=default.Table_4fdc1b366f04459b8b083811e9740ae1 [id=4ba4d4bb0a2d48b4aa7b956cafdb26a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.813539 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 205e951fef7445d79a900f256cff8074 (DEFAULT_TABLE table=default.Table_2541d743bd8945a2b0d984cdf911d6ba [id=825f8141b3ca4a1983f6548c66239504]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.814668 1856 ts_tablet_manager.cc:1950] T ca57963bcfb54eec9fcbed82757d8387 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:27.814815 746 tablet_service.cc:1505] Processing CreateTablet for tablet a6972d034dd54bd880dc09a3fb983d0c (DEFAULT_TABLE table=default.Table_4fdc1b366f04459b8b083811e9740ae1 [id=4ba4d4bb0a2d48b4aa7b956cafdb26a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.815260 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 353) took 1742 ms. Trace:
I20251214 20:52:27.815917 1982 rpcz_store.cc:276] 1214 20:52:26.072348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.073323 (+ 975us) service_pool.cc:225] Handling call
1214 20:52:27.815142 (+1741819us) ts_tablet_manager.cc:1151] Deleting tablet 57a6232564194cbba12182f3a29c2ebb
1214 20:52:27.815153 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.815241 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741550}
I20251214 20:52:27.820528 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205e951fef7445d79a900f256cff8074. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.821249 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 354) took 1517 ms. Trace:
I20251214 20:52:27.821620 1982 rpcz_store.cc:276] 1214 20:52:26.303744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.304029 (+ 285us) service_pool.cc:225] Handling call
1214 20:52:27.821129 (+1517100us) ts_tablet_manager.cc:1151] Deleting tablet e59b80ee919b4e1393fe317ed0e3a3d2
1214 20:52:27.821139 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.821233 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1505904}
I20251214 20:52:27.822289 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 355) took 1489 ms. Trace:
I20251214 20:52:27.822664 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6972d034dd54bd880dc09a3fb983d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.822621 1982 rpcz_store.cc:276] 1214 20:52:26.332401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.334076 (+ 1675us) service_pool.cc:225] Handling call
1214 20:52:27.822195 (+1488119us) ts_tablet_manager.cc:1151] Deleting tablet d745583012b4465090d85c75470e0bc2
1214 20:52:27.822204 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.822276 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1487876}
I20251214 20:52:27.825438 1203 tablet_bootstrap.cc:492] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:27.825693 1203 ts_tablet_manager.cc:1403] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.003s
I20251214 20:52:27.826539 1203 raft_consensus.cc:359] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.826823 1203 raft_consensus.cc:385] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.826920 1203 raft_consensus.cc:740] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.827229 1203 consensus_queue.cc:260] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.828100 1203 ts_tablet_manager.cc:1434] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:27.824572 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 356) took 1440 ms. Trace:
I20251214 20:52:27.829303 1982 rpcz_store.cc:276] 1214 20:52:26.384106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.384351 (+ 245us) service_pool.cc:225] Handling call
1214 20:52:27.823606 (+1439255us) ts_tablet_manager.cc:1151] Deleting tablet e7b247b761c94c60ad3a468db371b30b
1214 20:52:27.823618 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.824550 (+ 932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438929}
I20251214 20:52:27.829955 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 357) took 1332 ms. Trace:
I20251214 20:52:27.830093 1982 rpcz_store.cc:276] 1214 20:52:26.497072 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.497317 (+ 245us) service_pool.cc:225] Handling call
1214 20:52:27.829795 (+1332478us) ts_tablet_manager.cc:1151] Deleting tablet f1eb4c3aa654480caaa69dbd5047365c
1214 20:52:27.829805 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.829933 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1332219}
I20251214 20:52:27.830611 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 365) took 1078 ms. Trace:
I20251214 20:52:27.830747 1982 rpcz_store.cc:276] 1214 20:52:26.751910 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.751964 (+ 54us) service_pool.cc:225] Handling call
1214 20:52:27.830494 (+1078530us) ts_tablet_manager.cc:1151] Deleting tablet 4f358df4d9414dcdb0d197f434e28118
1214 20:52:27.830503 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.830596 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1078382}
I20251214 20:52:27.831236 1982 tablet_replica.cc:333] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:27.831524 1982 raft_consensus.cc:2243] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.834590 1982 raft_consensus.cc:2272] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.835502 1206 tablet_bootstrap.cc:654] T ba7952290e2c4746802cffeb14971da0 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.818149 883 tablet_service.cc:1505] Processing CreateTablet for tablet 205e951fef7445d79a900f256cff8074 (DEFAULT_TABLE table=default.Table_2541d743bd8945a2b0d984cdf911d6ba [id=825f8141b3ca4a1983f6548c66239504]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.818838 884 tablet_service.cc:1505] Processing CreateTablet for tablet a6972d034dd54bd880dc09a3fb983d0c (DEFAULT_TABLE table=default.Table_4fdc1b366f04459b8b083811e9740ae1 [id=4ba4d4bb0a2d48b4aa7b956cafdb26a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.819197 880 tablet_service.cc:1552] Processing DeleteTablet for tablet d171301941fa43908e05984c3b016007 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.829172 2564 raft_consensus.cc:493] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.836577 2564 raft_consensus.cc:515] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.836970 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6972d034dd54bd880dc09a3fb983d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.837555 2564 leader_election.cc:290] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:27.836047 848 consensus_peers.cc:597] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebb26a5ae7224e9ca8a513c7b5013618. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:27.839334 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205e951fef7445d79a900f256cff8074. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.840664 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17550eb0ee6b476f8a8c304451f678b2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:27.840973 1050 raft_consensus.cc:2468] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:27.841365 1856 tablet_replica.cc:333] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:27.841610 1856 raft_consensus.cc:2243] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.841768 1856 raft_consensus.cc:2272] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.842382 851 leader_election.cc:304] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:27.842432 1856 ts_tablet_manager.cc:1916] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.842775 2564 raft_consensus.cc:2804] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:27.842852 2564 raft_consensus.cc:493] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:27.842947 2564 raft_consensus.cc:3060] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:27.843947 2571 raft_consensus.cc:493] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.844075 2571 raft_consensus.cc:515] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.844815 2571 leader_election.cc:290] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:27.846264 2564 raft_consensus.cc:515] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:27.847286 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c9488c3505f5443998531cf749722be9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9488c3505f5443998531cf749722be9 already in progress: creating tablet
W20251214 20:52:27.847890 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 57a6232564194cbba12182f3a29c2ebb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57a6232564194cbba12182f3a29c2ebb already in progress: creating tablet
I20251214 20:52:27.847919 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47e813373fbd482298312ead4f037c01" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:27.848065 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]) successfully deleted
W20251214 20:52:27.856320 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet e59b80ee919b4e1393fe317ed0e3a3d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e59b80ee919b4e1393fe317ed0e3a3d2 already in progress: creating tablet
W20251214 20:52:27.856698 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet d745583012b4465090d85c75470e0bc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d745583012b4465090d85c75470e0bc2 already in progress: creating tablet
W20251214 20:52:27.857324 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet e7b247b761c94c60ad3a468db371b30b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7b247b761c94c60ad3a468db371b30b already in progress: creating tablet
W20251214 20:52:27.857559 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet f1eb4c3aa654480caaa69dbd5047365c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1eb4c3aa654480caaa69dbd5047365c already in progress: creating tablet
W20251214 20:52:27.857933 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 4f358df4d9414dcdb0d197f434e28118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f358df4d9414dcdb0d197f434e28118 already in progress: creating tablet
W20251214 20:52:27.848786 851 leader_election.cc:343] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:27.858264 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 16816fa6b8c34207bb929147910b93cb because the tablet was not found. No further retry: Not found: Tablet not found: 16816fa6b8c34207bb929147910b93cb
W20251214 20:52:27.858493 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 5f4760585baf4b4ab3b9b94b5a748a61 because the tablet was not found. No further retry: Not found: Tablet not found: 5f4760585baf4b4ab3b9b94b5a748a61
I20251214 20:52:27.858451 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17550eb0ee6b476f8a8c304451f678b2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:27.858613 1050 raft_consensus.cc:3060] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:27.860524 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2accf8a160584c82bb695a52e3f2b05c because the tablet was not found. No further retry: Not found: Tablet not found: 2accf8a160584c82bb695a52e3f2b05c
I20251214 20:52:27.861035 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ac56c984d6f5476cb187409e4e7575b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:27.861275 1050 raft_consensus.cc:2468] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:27.861383 1856 ts_tablet_manager.cc:1929] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:27.861527 1856 log.cc:1199] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ec85857f047e44d78835305ddcb70768
I20251214 20:52:27.861707 851 leader_election.cc:304] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:27.861850 1856 ts_tablet_manager.cc:1950] T ec85857f047e44d78835305ddcb70768 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:27.861843 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac56c984d6f5476cb187409e4e7575b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.862512 2564 leader_election.cc:290] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:27.862695 2564 raft_consensus.cc:2804] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:27.863207 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet ec85857f047e44d78835305ddcb70768 (table default.Table_58a7d5bd289f408da592c143483b0329 [id=bb060058712440c7b1e5c40332a0263d]) successfully deleted
I20251214 20:52:27.863265 2564 raft_consensus.cc:697] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:27.863327 1856 tablet_replica.cc:333] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:27.863534 1856 raft_consensus.cc:2243] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:27.863516 2564 consensus_queue.cc:237] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.864014 1856 raft_consensus.cc:2272] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.865175 1856 ts_tablet_manager.cc:1916] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:27.865247 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b5360e7ece94fd2a5a6512db6c6c12f has no consensus state
I20251214 20:52:27.866667 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ee3a88061eb34c19b3fad30276bc7ece" } new_table_name: "default.Table_d59e2b0d82384da68c1901ddaa419d00" modify_external_catalogs: true
I20251214 20:52:27.869728 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_52200ef9aa154f64ae25af61268f6d6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.870747 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52200ef9aa154f64ae25af61268f6d6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.878978 880 tablet_service.cc:1505] Processing CreateTablet for tablet 6be0a51f763b474093cc7d08ea75411e (DEFAULT_TABLE table=default.Table_5126357c033645d1ae06a909124ab640 [id=72c5a8ce76cb46d89b7a563f8b16abc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.880153 1856 ts_tablet_manager.cc:1929] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.881700 1856 log.cc:1199] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/98105bceda9b44bf893f82a4234a3c81
I20251214 20:52:27.882187 1856 ts_tablet_manager.cc:1950] T 98105bceda9b44bf893f82a4234a3c81 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:27.883805 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 6be0a51f763b474093cc7d08ea75411e (DEFAULT_TABLE table=default.Table_5126357c033645d1ae06a909124ab640 [id=72c5a8ce76cb46d89b7a563f8b16abc3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:27.888679 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2807af0754914688bd7a090b1bc32eb1 consensus state has no leader
I20251214 20:52:27.889088 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b971804db7ad428a88f22d83a1de29d9" } modify_external_catalogs: true
I20251214 20:52:27.890194 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 419) took 1044 ms. Trace:
I20251214 20:52:27.890317 453 master_service.cc:954] ReplaceTablet: received request to replace tablet c3453fc996fb4d2b8cf0f80152098c43 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:27.892309 1856 rpcz_store.cc:276] 1214 20:52:26.845526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.941079 (+ 95553us) service_pool.cc:225] Handling call
1214 20:52:27.863252 (+922173us) ts_tablet_manager.cc:1151] Deleting tablet 98105bceda9b44bf893f82a4234a3c81
1214 20:52:27.863263 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.875928 (+ 12665us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.880027 (+ 4099us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.890160 (+ 10133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1974,"tablet-delete.queue_time_us":908253}
I20251214 20:52:27.892885 887 tablet_service.cc:1505] Processing CreateTablet for tablet f5f6cbcb80e4428898148180080273c8 (DEFAULT_TABLE table=default.Table_94c723f6fee24f40a5787f7e5e92d03f [id=223e7226cfa94865a38476e84504a946]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.894958 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 17550eb0ee6b476f8a8c304451f678b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.895246 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet ca57963bcfb54eec9fcbed82757d8387 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.895241 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b1a937150ce4ecfa210c8465d682496 with delete_type TABLET_DATA_DELETED (replaced by tablet f1eb4c3aa654480caaa69dbd5047365c) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.895618 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6be0a51f763b474093cc7d08ea75411e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.897605 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_b971804db7ad428a88f22d83a1de29d9" } modify_external_catalogs: true
I20251214 20:52:27.897907 451 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_fe0d773656e24c0e9d6e2e36caeaa9f8 [id=ed335fb680bc441184e1682200ee68be] alter complete (version 3)
I20251214 20:52:27.899097 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 17550eb0ee6b476f8a8c304451f678b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.899720 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_c3bedc14bbf14c5988f9c60ceacf9d95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:27.899930 1856 tablet_replica.cc:333] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:27.900034 877 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c973e0111804b8b869068bdcc5869bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.900166 1856 raft_consensus.cc:2243] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.900336 1856 raft_consensus.cc:2272] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:27.900336 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3bedc14bbf14c5988f9c60ceacf9d95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.900676 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 37a33cc9bb784f72956ac9f5493a0c3d with delete_type TABLET_DATA_DELETED (replaced by tablet b87b0871c2454971a15dd5643c155694) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.900761 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d164747cb294bd4b126b4420231867d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.900887 1856 ts_tablet_manager.cc:1916] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.900924 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d6e7306254044449f1127fb963ca5f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:27.901971 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_0f9951606583457cb248ab0e617b466b" } modify_external_catalogs: true
I20251214 20:52:27.902134 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_0f9951606583457cb248ab0e617b466b" } modify_external_catalogs: true
I20251214 20:52:27.903264 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_694f6a7aab6f4c76b248dd05c441ea03" } new_table_name: "default.Table_4649d8b3e43c42a492723fadd4217e4e" modify_external_catalogs: true
I20251214 20:52:27.904440 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]) successfully deleted
W20251214 20:52:27.904717 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 492770a7b8b74ba49315f974cc54eedb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 492770a7b8b74ba49315f974cc54eedb already in progress: creating tablet
I20251214 20:52:27.904639 2551 consensus_queue.cc:1048] T 190afbb866854dd7b5393e04443752b7 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:27.904922 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 6a000951f61948ac85fdd431f3d162c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a000951f61948ac85fdd431f3d162c2 already in progress: creating tablet
W20251214 20:52:27.905103 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 37a33cc9bb784f72956ac9f5493a0c3d because the tablet was not found. No further retry: Not found: Tablet not found: 37a33cc9bb784f72956ac9f5493a0c3d
I20251214 20:52:27.907755 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_35def73a244b4885ad22656a1df2b697"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.908511 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35def73a244b4885ad22656a1df2b697 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.912544 879 tablet_service.cc:1505] Processing CreateTablet for tablet a6517a7974cd430e95d976fa3f99e465 (DEFAULT_TABLE table=default.Table_d9babf534d4b4411aa03906984198232 [id=2e2d38b971be42a1befe0d6995030970]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.913420 1203 tablet_bootstrap.cc:492] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:27.914853 885 tablet_service.cc:1505] Processing CreateTablet for tablet a4bcac304fd14e20b0607c23d1569a31 (DEFAULT_TABLE table=default.Table_d8bb40c2324c4c0eb7e52aa96e920813 [id=3b04d86591b14695a334b68b0572f371]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.915203 1203 tablet_bootstrap.cc:654] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:27.916455 1018 tablet_service.cc:1505] Processing CreateTablet for tablet a6517a7974cd430e95d976fa3f99e465 (DEFAULT_TABLE table=default.Table_d9babf534d4b4411aa03906984198232 [id=2e2d38b971be42a1befe0d6995030970]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.917182 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6517a7974cd430e95d976fa3f99e465. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.917801 2551 raft_consensus.cc:493] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.917958 2551 raft_consensus.cc:515] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.918562 2551 leader_election.cc:290] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:27.919251 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9da5e98c6f4843449886176e9fcfa8e1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:27.920006 991 leader_election.cc:343] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.836385 1982 ts_tablet_manager.cc:1916] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:27.926558 1028 tablet_service.cc:1505] Processing CreateTablet for tablet a4bcac304fd14e20b0607c23d1569a31 (DEFAULT_TABLE table=default.Table_d8bb40c2324c4c0eb7e52aa96e920813 [id=3b04d86591b14695a334b68b0572f371]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.926885 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6517a7974cd430e95d976fa3f99e465. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.926947 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4bcac304fd14e20b0607c23d1569a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.839519 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17550eb0ee6b476f8a8c304451f678b2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:27.846992 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47e813373fbd482298312ead4f037c01" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:27.927256 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4bcac304fd14e20b0607c23d1569a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.857834 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17550eb0ee6b476f8a8c304451f678b2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:27.927923 848 leader_election.cc:343] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.928033 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205e951fef7445d79a900f256cff8074. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:27.928263 848 leader_election.cc:343] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.928359 848 leader_election.cc:304] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:27.928622 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f6cbcb80e4428898148180080273c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:27.928731 848 leader_election.cc:343] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.892654 2468 raft_consensus.cc:493] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.929057 2468 raft_consensus.cc:515] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.929817 2468 leader_election.cc:290] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:27.897630 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 2accf8a160584c82bb695a52e3f2b05c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.897572 748 tablet_service.cc:1505] Processing CreateTablet for tablet f5f6cbcb80e4428898148180080273c8 (DEFAULT_TABLE table=default.Table_94c723f6fee24f40a5787f7e5e92d03f [id=223e7226cfa94865a38476e84504a946]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.930270 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f6cbcb80e4428898148180080273c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:27.930572 712 leader_election.cc:343] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: b315744c8cd043c5a8bdceec73a7e28b
I20251214 20:52:27.930648 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6972d034dd54bd880dc09a3fb983d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.931629 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6be0a51f763b474093cc7d08ea75411e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.932941 1982 ts_tablet_manager.cc:1929] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:27.933044 1982 log.cc:1199] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ca57963bcfb54eec9fcbed82757d8387
I20251214 20:52:27.933384 1982 ts_tablet_manager.cc:1950] T ca57963bcfb54eec9fcbed82757d8387 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:27.933730 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 371) took 1108 ms. Trace:
I20251214 20:52:27.933806 1982 rpcz_store.cc:276] 1214 20:52:26.825145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.825265 (+ 120us) service_pool.cc:225] Handling call
1214 20:52:27.831119 (+1005854us) ts_tablet_manager.cc:1151] Deleting tablet ca57963bcfb54eec9fcbed82757d8387
1214 20:52:27.831194 (+ 75us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.929675 (+ 98481us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.932833 (+ 3158us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.933712 (+ 879us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005549}
I20251214 20:52:27.897578 743 tablet_service.cc:1552] Processing DeleteTablet for tablet b315744c8cd043c5a8bdceec73a7e28b with delete_type TABLET_DATA_DELETED (replaced by tablet f945814952b14131ba4e0c67a403a680) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.912067 747 tablet_service.cc:1505] Processing CreateTablet for tablet a6517a7974cd430e95d976fa3f99e465 (DEFAULT_TABLE table=default.Table_d9babf534d4b4411aa03906984198232 [id=2e2d38b971be42a1befe0d6995030970]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.914212 749 tablet_service.cc:1505] Processing CreateTablet for tablet a4bcac304fd14e20b0607c23d1569a31 (DEFAULT_TABLE table=default.Table_d8bb40c2324c4c0eb7e52aa96e920813 [id=3b04d86591b14695a334b68b0572f371]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.919544 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9da5e98c6f4843449886176e9fcfa8e1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:27.935503 448 master_service.cc:954] ReplaceTablet: received request to replace tablet a0976ffcc649461fac74e20b5e402275 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:27.936921 743 tablet_service.cc:1505] Processing CreateTablet for tablet 44c61824bacd406280ba03357f9f5b1d (DEFAULT_TABLE table=default.Table_47862fea18504c61af1783e5d8fca95e [id=4ccf39e7c864479eba17718e3d08afe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.937214 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6517a7974cd430e95d976fa3f99e465. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.937383 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet c3453fc996fb4d2b8cf0f80152098c43
I20251214 20:52:27.937556 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c61824bacd406280ba03357f9f5b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.937878 453 catalog_manager.cc:6724] ReplaceTablet: tablet c3453fc996fb4d2b8cf0f80152098c43 deleted and replaced by tablet ba322cb6ee084bc39788ddd439f5f15d
I20251214 20:52:27.938165 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_93fe3196f8c94960a5c5ee9e1f3923f1" } modify_external_catalogs: true
I20251214 20:52:27.938311 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_93fe3196f8c94960a5c5ee9e1f3923f1" } modify_external_catalogs: true
W20251214 20:52:27.940059 989 leader_election.cc:343] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:27.940179 989 leader_election.cc:304] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:27.940713 2551 raft_consensus.cc:2749] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.942451 452 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 9da5e98c6f4843449886176e9fcfa8e1 (table default.Table_55f303d33ec84a269232fc1ba8f2d182 [id=5ded9fe825ae4308b5bd4b21c16634c8]). Expected version 1 got 0
I20251214 20:52:27.943305 2571 raft_consensus.cc:2749] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.943406 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4bcac304fd14e20b0607c23d1569a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.943810 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_eba09dc20b824904b171bc39be41ea65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:27.944571 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eba09dc20b824904b171bc39be41ea65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:27.945112 745 tablet_service.cc:1552] Processing DeleteTablet for tablet c3453fc996fb4d2b8cf0f80152098c43 with delete_type TABLET_DATA_DELETED (replaced by tablet ba322cb6ee084bc39788ddd439f5f15d) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.944933 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]) successfully deleted
W20251214 20:52:27.946197 713 leader_election.cc:343] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: b315744c8cd043c5a8bdceec73a7e28b
I20251214 20:52:27.946308 713 leader_election.cc:304] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:27.945221 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a0f669313244e809c2868ca0ccfad65 consensus state has no leader
I20251214 20:52:27.946686 2468 raft_consensus.cc:2749] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.946765 449 master_service.cc:954] ReplaceTablet: received request to replace tablet 0a2fbec0543445b7856ef22c934c6840 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:27.951048 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 367) took 1126 ms. Trace:
I20251214 20:52:27.951751 1982 rpcz_store.cc:276] 1214 20:52:26.824765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.825476 (+ 711us) service_pool.cc:225] Handling call
1214 20:52:27.934091 (+1108615us) ts_tablet_manager.cc:1151] Deleting tablet 9d89ee602c524c96a21c0a235af07481
1214 20:52:27.934105 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.951025 (+ 16920us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1108412}
I20251214 20:52:27.953500 1982 tablet_replica.cc:333] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:27.954154 1982 raft_consensus.cc:2243] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:27.954909 1982 raft_consensus.cc:2272] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:27.955978 1982 ts_tablet_manager.cc:1916] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:27.960362 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 9d89ee602c524c96a21c0a235af07481 because the tablet was not found. No further retry: Not found: Tablet not found: 9d89ee602c524c96a21c0a235af07481
I20251214 20:52:27.961087 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet c3453fc996fb4d2b8cf0f80152098c43 with delete_type TABLET_DATA_DELETED (replaced by tablet ba322cb6ee084bc39788ddd439f5f15d) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.961339 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ac06f852fc4abea5cb213926002ca1 with delete_type TABLET_DATA_DELETED (replaced by tablet 248e3666fd13434e97378eb5a98dfadd) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.963294 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:27.964350 1202 tablet_bootstrap.cc:492] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:27.964634 1202 ts_tablet_manager.cc:1403] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.201s user 0.004s sys 0.001s
I20251214 20:52:27.965579 1202 raft_consensus.cc:359] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.965937 1202 raft_consensus.cc:385] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:27.966032 1202 raft_consensus.cc:740] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:27.966346 1202 consensus_queue.cc:260] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.968242 1202 ts_tablet_manager.cc:1434] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:27.966655 1977 ts_tablet_manager.cc:1929] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.969501 1977 log.cc:1199] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ac64f4fcb6a943668584658df19f950c
I20251214 20:52:27.969909 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 44c61824bacd406280ba03357f9f5b1d (DEFAULT_TABLE table=default.Table_47862fea18504c61af1783e5d8fca95e [id=4ccf39e7c864479eba17718e3d08afe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.970532 1977 ts_tablet_manager.cc:1950] T ac64f4fcb6a943668584658df19f950c P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:27.970992 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c61824bacd406280ba03357f9f5b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:27.972177 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 316) took 3775 ms. Trace:
I20251214 20:52:27.977144 2468 raft_consensus.cc:493] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:27.977298 2468 raft_consensus.cc:515] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:27.978005 2468 leader_election.cc:290] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:27.979086 713 leader_election.cc:343] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 2d164747cb294bd4b126b4420231867d
W20251214 20:52:27.979373 712 leader_election.cc:343] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 2d164747cb294bd4b126b4420231867d
I20251214 20:52:27.979494 712 leader_election.cc:304] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:27.979720 2468 raft_consensus.cc:2749] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:27.987771 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d164747cb294bd4b126b4420231867d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.988317 743 tablet_service.cc:1505] Processing CreateTablet for tablet c8e43456af5449939107def570689b7c (DEFAULT_TABLE table=default.Table_cb1220aeec334a3c8aa15d0049c58d75 [id=3e833b0eaf3a4f519a1c1cfebaa0e969]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:27.988394 749 tablet_service.cc:1552] Processing DeleteTablet for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:27.992877 2549 tablet.cc:1722] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:27.994959 1982 ts_tablet_manager.cc:1929] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:27.997224 451 master_service.cc:954] ReplaceTablet: received request to replace tablet 825baba561ee4a8ebd44f75d8c51706a from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:27.997893 454 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 7e0847d58c0d4c5cbd1f9b91fd103763
I20251214 20:52:27.999161 1023 tablet_service.cc:1505] Processing CreateTablet for tablet c8e43456af5449939107def570689b7c (DEFAULT_TABLE table=default.Table_cb1220aeec334a3c8aa15d0049c58d75 [id=3e833b0eaf3a4f519a1c1cfebaa0e969]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.000236 1203 tablet_bootstrap.cc:492] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:28.000423 1203 ts_tablet_manager.cc:1403] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.005s
I20251214 20:52:28.001204 1203 raft_consensus.cc:359] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.001399 1203 raft_consensus.cc:385] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.001451 1203 raft_consensus.cc:740] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.001559 2468 consensus_queue.cc:1048] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:28.001700 1203 consensus_queue.cc:260] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.002329 1203 ts_tablet_manager.cc:1434] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:28.010176 2550 tablet.cc:1722] T 65b2fe0de22646e4b77fbc6a289ff4a1 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:28.012490 1977 rpcz_store.cc:276] 1214 20:52:24.197024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.197074 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:27.751242 (+3554168us) ts_tablet_manager.cc:1151] Deleting tablet ac64f4fcb6a943668584658df19f950c
1214 20:52:27.751254 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.930768 (+179514us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.966527 (+ 35759us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.972154 (+ 5627us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3524603}
I20251214 20:52:28.013221 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 319) took 3728 ms. Trace:
I20251214 20:52:28.013366 1977 rpcz_store.cc:276] 1214 20:52:24.285246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.285334 (+ 88us) service_pool.cc:225] Handling call
1214 20:52:28.013099 (+3727765us) ts_tablet_manager.cc:1151] Deleting tablet 054cfd587d19409fbdf7f4848bcaf3f9
1214 20:52:28.013109 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.013205 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3705079}
I20251214 20:52:27.976778 877 tablet_service.cc:1505] Processing CreateTablet for tablet 44c61824bacd406280ba03357f9f5b1d (DEFAULT_TABLE table=default.Table_47862fea18504c61af1783e5d8fca95e [id=4ccf39e7c864479eba17718e3d08afe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.014003 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 318) took 3731 ms. Trace:
I20251214 20:52:28.015435 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ac64f4fcb6a943668584658df19f950c (table default.Table_5275181d46dc4a85b27fc0c45fa587fe [id=ab9838348eb24201bcb2866b16994965]) successfully deleted
W20251214 20:52:28.015838 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 054cfd587d19409fbdf7f4848bcaf3f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 054cfd587d19409fbdf7f4848bcaf3f9 already in progress: creating tablet
I20251214 20:52:28.015926 1856 ts_tablet_manager.cc:1929] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:27.992813 2574 tablet.cc:1722] T 65b2fe0de22646e4b77fbc6a289ff4a1 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:28.018185 1977 rpcz_store.cc:276] 1214 20:52:24.282872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.285324 (+ 2452us) service_pool.cc:225] Handling call
1214 20:52:28.013906 (+3728582us) ts_tablet_manager.cc:1151] Deleting tablet 43fe3bb1ed3941ce9078376f8f478408
1214 20:52:28.013915 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.013986 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3704247}
I20251214 20:52:28.018607 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 324) took 3698 ms. Trace:
I20251214 20:52:28.018736 1977 rpcz_store.cc:276] 1214 20:52:24.319691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.319775 (+ 84us) service_pool.cc:225] Handling call
1214 20:52:28.018486 (+3698711us) ts_tablet_manager.cc:1151] Deleting tablet 727e7b61ae3942098712720d344c5310
1214 20:52:28.018497 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.018592 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3698236}
I20251214 20:52:28.019085 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 343) took 3207 ms. Trace:
W20251214 20:52:28.019085 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 43fe3bb1ed3941ce9078376f8f478408 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43fe3bb1ed3941ce9078376f8f478408 already in progress: creating tablet
I20251214 20:52:28.019217 1977 rpcz_store.cc:276] 1214 20:52:24.811242 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.811315 (+ 73us) service_pool.cc:225] Handling call
1214 20:52:28.018992 (+3207677us) ts_tablet_manager.cc:1151] Deleting tablet 54e5fff1fc84419ab3d3131cb3a12d92
1214 20:52:28.019001 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.019071 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3207483}
I20251214 20:52:28.019459 1977 tablet_replica.cc:333] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:28.019742 1977 raft_consensus.cc:2243] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:28.019924 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 727e7b61ae3942098712720d344c5310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 727e7b61ae3942098712720d344c5310 already in progress: creating tablet
I20251214 20:52:28.019989 1977 raft_consensus.cc:2272] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:28.020258 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 54e5fff1fc84419ab3d3131cb3a12d92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54e5fff1fc84419ab3d3131cb3a12d92 already in progress: creating tablet
I20251214 20:52:28.010596 1202 tablet_bootstrap.cc:492] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:27.976325 878 tablet_service.cc:1552] Processing DeleteTablet for tablet c3453fc996fb4d2b8cf0f80152098c43 with delete_type TABLET_DATA_DELETED (replaced by tablet ba322cb6ee084bc39788ddd439f5f15d) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.020380 1856 log.cc:1199] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/b2e83bca8ed642b6a1b705ef70fde27f
I20251214 20:52:28.014037 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c61824bacd406280ba03357f9f5b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.020887 1856 ts_tablet_manager.cc:1950] T b2e83bca8ed642b6a1b705ef70fde27f P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:28.021206 2523 consensus_queue.cc:1048] T 6fef1d6add764fd28a8f4995b337ea71 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:27.995097 1982 log.cc:1199] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/16816fa6b8c34207bb929147910b93cb
I20251214 20:52:28.022441 1982 ts_tablet_manager.cc:1950] T 16816fa6b8c34207bb929147910b93cb P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:27.988169 890 tablet_service.cc:1505] Processing CreateTablet for tablet c8e43456af5449939107def570689b7c (DEFAULT_TABLE table=default.Table_cb1220aeec334a3c8aa15d0049c58d75 [id=3e833b0eaf3a4f519a1c1cfebaa0e969]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.023496 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e43456af5449939107def570689b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.024755 1202 tablet_bootstrap.cc:654] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.028028 1206 tablet_bootstrap.cc:492] T ba7952290e2c4746802cffeb14971da0 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:28.028282 1206 ts_tablet_manager.cc:1403] T ba7952290e2c4746802cffeb14971da0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.253s user 0.003s sys 0.002s
I20251214 20:52:28.029268 1206 raft_consensus.cc:359] T ba7952290e2c4746802cffeb14971da0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.029560 1206 raft_consensus.cc:385] T ba7952290e2c4746802cffeb14971da0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.029659 1206 raft_consensus.cc:740] T ba7952290e2c4746802cffeb14971da0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.029946 1206 consensus_queue.cc:260] T ba7952290e2c4746802cffeb14971da0 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.030686 1206 ts_tablet_manager.cc:1434] T ba7952290e2c4746802cffeb14971da0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:28.031529 1977 ts_tablet_manager.cc:1916] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.039484 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e43456af5449939107def570689b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.039934 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 368) took 1215 ms. Trace:
I20251214 20:52:28.040025 1982 rpcz_store.cc:276] 1214 20:52:26.824881 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.825720 (+ 839us) service_pool.cc:225] Handling call
1214 20:52:27.953441 (+1127721us) ts_tablet_manager.cc:1151] Deleting tablet 16816fa6b8c34207bb929147910b93cb
1214 20:52:27.953451 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:27.991888 (+ 38437us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:27.994846 (+ 2958us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.039914 (+ 45068us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127590}
I20251214 20:52:28.041450 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 369) took 1216 ms. Trace:
I20251214 20:52:28.041646 1982 rpcz_store.cc:276] 1214 20:52:26.824961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.825868 (+ 907us) service_pool.cc:225] Handling call
1214 20:52:28.040459 (+1214591us) ts_tablet_manager.cc:1151] Deleting tablet 4d6e7306254044449f1127fb963ca5f0
1214 20:52:28.040468 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.041429 (+ 961us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1214484}
I20251214 20:52:28.042017 1982 tablet_replica.cc:333] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:28.042275 1982 raft_consensus.cc:2243] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.042524 1982 raft_consensus.cc:2272] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.043756 1206 tablet_bootstrap.cc:492] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:28.044773 1977 ts_tablet_manager.cc:1929] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:28.045832 1977 log.cc:1199] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/7d4626d02ca548f3bf5a1a9142600bff
I20251214 20:52:28.045907 1206 tablet_bootstrap.cc:654] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.046396 1977 ts_tablet_manager.cc:1950] T 7d4626d02ca548f3bf5a1a9142600bff P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:28.047431 1982 ts_tablet_manager.cc:1916] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.047641 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 16816fa6b8c34207bb929147910b93cb (table default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]) successfully deleted
I20251214 20:52:28.047868 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 344) took 3213 ms. Trace:
W20251214 20:52:28.048099 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 4d6e7306254044449f1127fb963ca5f0 because the tablet was not found. No further retry: Not found: Tablet not found: 4d6e7306254044449f1127fb963ca5f0
I20251214 20:52:28.048822 1977 rpcz_store.cc:276] 1214 20:52:24.834101 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.834384 (+ 283us) service_pool.cc:225] Handling call
1214 20:52:28.019415 (+3185031us) ts_tablet_manager.cc:1151] Deleting tablet 7d4626d02ca548f3bf5a1a9142600bff
1214 20:52:28.019423 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.034740 (+ 15317us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.044639 (+ 9899us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.047845 (+ 3206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3155029}
I20251214 20:52:28.049654 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 7d4626d02ca548f3bf5a1a9142600bff (table default.Table_6d9954b482a4427ba8a996c085f831bf [id=d402227da94f418bbed00186b8ac48c9]) successfully deleted
I20251214 20:52:28.050067 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 345) took 3089 ms. Trace:
I20251214 20:52:28.050274 1977 rpcz_store.cc:276] 1214 20:52:24.960585 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:24.960640 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:28.049652 (+3089012us) ts_tablet_manager.cc:1151] Deleting tablet 57f2314d33fa474ca36dafe0a0eae5ae
1214 20:52:28.049664 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.050048 (+ 384us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3088809}
W20251214 20:52:28.051026 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 57f2314d33fa474ca36dafe0a0eae5ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57f2314d33fa474ca36dafe0a0eae5ae already in progress: creating tablet
I20251214 20:52:28.051854 1977 tablet_replica.cc:333] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:28.052160 1977 raft_consensus.cc:2243] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:28.052951 1977 raft_consensus.cc:2272] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.053555 1982 ts_tablet_manager.cc:1929] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:28.054499 1977 ts_tablet_manager.cc:1916] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.068394 1977 ts_tablet_manager.cc:1929] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:28.069041 1977 log.cc:1199] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ebb26a5ae7224e9ca8a513c7b5013618
I20251214 20:52:28.069834 1022 tablet_service.cc:1505] Processing CreateTablet for tablet f5f6cbcb80e4428898148180080273c8 (DEFAULT_TABLE table=default.Table_94c723f6fee24f40a5787f7e5e92d03f [id=223e7226cfa94865a38476e84504a946]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.069984 1977 ts_tablet_manager.cc:1950] T ebb26a5ae7224e9ca8a513c7b5013618 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:28.073597 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 346) took 3052 ms. Trace:
I20251214 20:52:28.073899 1977 rpcz_store.cc:276] 1214 20:52:25.021453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.025653 (+ 4200us) service_pool.cc:225] Handling call
1214 20:52:28.051459 (+3025806us) ts_tablet_manager.cc:1151] Deleting tablet ebb26a5ae7224e9ca8a513c7b5013618
1214 20:52:28.051468 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.057794 (+ 6326us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.068262 (+ 10468us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.073572 (+ 5310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2954667}
I20251214 20:52:28.076226 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ebb26a5ae7224e9ca8a513c7b5013618 (table default.Table_4907bb640d114586b98e6fa0e514cb5a [id=2a6d9d4658f24eb5b39adea101a0891c]) successfully deleted
I20251214 20:52:28.076726 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 347) took 2724 ms. Trace:
I20251214 20:52:28.076869 1977 rpcz_store.cc:276] 1214 20:52:25.351985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.352041 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:28.076597 (+2724556us) ts_tablet_manager.cc:1151] Deleting tablet 248e3666fd13434e97378eb5a98dfadd
1214 20:52:28.076609 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.076710 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2724328}
W20251214 20:52:28.077407 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 248e3666fd13434e97378eb5a98dfadd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 248e3666fd13434e97378eb5a98dfadd already in progress: creating tablet
I20251214 20:52:28.077821 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 348) took 2707 ms. Trace:
I20251214 20:52:28.077932 1977 rpcz_store.cc:276] 1214 20:52:25.370395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.370460 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:28.077707 (+2707247us) ts_tablet_manager.cc:1151] Deleting tablet e6ebfc373fbf41b1920c271a15302484
1214 20:52:28.077717 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.077806 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2707037}
W20251214 20:52:28.078364 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet e6ebfc373fbf41b1920c271a15302484 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6ebfc373fbf41b1920c271a15302484 already in progress: creating tablet
I20251214 20:52:28.078666 1977 tablet_replica.cc:333] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:28.078893 1977 raft_consensus.cc:2243] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.079301 1977 raft_consensus.cc:2272] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.080075 1977 ts_tablet_manager.cc:1916] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.087304 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e43456af5449939107def570689b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.087510 1203 tablet_bootstrap.cc:492] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:28.089254 1203 tablet_bootstrap.cc:654] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.091315 1856 tablet_replica.cc:333] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:28.091554 1856 raft_consensus.cc:2243] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.091733 1856 raft_consensus.cc:2272] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:28.092168 848 consensus_peers.cc:597] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:28.092296 1856 ts_tablet_manager.cc:1916] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.095686 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet b2e83bca8ed642b6a1b705ef70fde27f (table default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7]) successfully deleted
I20251214 20:52:28.096451 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f6cbcb80e4428898148180080273c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.099965 2551 raft_consensus.cc:493] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.100112 2551 raft_consensus.cc:515] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.100732 2551 leader_election.cc:290] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.102224 1977 ts_tablet_manager.cc:1929] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:28.103750 1977 log.cc:1199] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/d1389716f5ce44f588c21a34c7a7d4c8
I20251214 20:52:28.104226 1977 ts_tablet_manager.cc:1950] T d1389716f5ce44f588c21a34c7a7d4c8 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:28.104926 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbbfcef827ee4d548939aaf4783c8109" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.105113 915 raft_consensus.cc:2468] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:28.105579 991 leader_election.cc:304] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.105844 2551 raft_consensus.cc:2804] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:28.105916 2551 raft_consensus.cc:493] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:28.105981 2551 raft_consensus.cc:3060] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.106208 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 349) took 2565 ms. Trace:
I20251214 20:52:28.106500 1977 rpcz_store.cc:276] 1214 20:52:25.541044 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.541115 (+ 71us) service_pool.cc:225] Handling call
1214 20:52:28.078621 (+2537506us) ts_tablet_manager.cc:1151] Deleting tablet d1389716f5ce44f588c21a34c7a7d4c8
1214 20:52:28.078630 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.083892 (+ 5262us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.102134 (+ 18242us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.106191 (+ 4057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2202281}
I20251214 20:52:28.107421 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet d1389716f5ce44f588c21a34c7a7d4c8 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]) successfully deleted
I20251214 20:52:28.107873 1977 tablet_replica.cc:333] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:28.108269 1977 raft_consensus.cc:2243] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.109078 2551 raft_consensus.cc:515] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.110178 2551 leader_election.cc:290] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.115111 1856 ts_tablet_manager.cc:1929] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:28.054230 1206 tablet_bootstrap.cc:492] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:28.116168 1206 ts_tablet_manager.cc:1403] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.005s
I20251214 20:52:28.116519 1856 log.cc:1199] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/69f6108792fa4c2a9744e3248c76251f
I20251214 20:52:28.116865 1206 raft_consensus.cc:359] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.117026 1206 raft_consensus.cc:385] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.117072 1206 raft_consensus.cc:740] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.117255 1206 consensus_queue.cc:260] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.117743 1206 ts_tablet_manager.cc:1434] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251214 20:52:28.118127 1982 log.cc:1199] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ca263a80702f4f64aa7c872c2be08207
I20251214 20:52:28.118181 1206 tablet_bootstrap.cc:492] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:28.118561 1982 ts_tablet_manager.cc:1950] T ca263a80702f4f64aa7c872c2be08207 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:28.087613 746 tablet_service.cc:1505] Processing CreateTablet for tablet 6be0a51f763b474093cc7d08ea75411e (DEFAULT_TABLE table=default.Table_5126357c033645d1ae06a909124ab640 [id=72c5a8ce76cb46d89b7a563f8b16abc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.119906 1206 tablet_bootstrap.cc:654] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.122495 1977 raft_consensus.cc:2272] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.127565 1206 tablet_bootstrap.cc:492] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:28.128096 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_87f9c82a6a89490391b5c78001639d2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:28.128316 1206 ts_tablet_manager.cc:1403] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20251214 20:52:28.128682 2561 raft_consensus.cc:493] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:28.128726 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87f9c82a6a89490391b5c78001639d2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.128384 1856 ts_tablet_manager.cc:1950] T 69f6108792fa4c2a9744e3248c76251f P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:28.128818 2561 raft_consensus.cc:515] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.129381 2561 leader_election.cc:290] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.129320 1206 raft_consensus.cc:359] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.129501 1206 raft_consensus.cc:385] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.129554 1206 raft_consensus.cc:740] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.129997 1206 consensus_queue.cc:260] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.130527 1206 ts_tablet_manager.cc:1434] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:28.132772 1206 tablet_bootstrap.cc:492] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:28.134294 1206 tablet_bootstrap.cc:654] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.138486 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 370) took 1313 ms. Trace:
I20251214 20:52:28.138805 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6be0a51f763b474093cc7d08ea75411e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.110481 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbbfcef827ee4d548939aaf4783c8109" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:28.130800 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58ac06f852fc4abea5cb213926002ca1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.139403 910 raft_consensus.cc:2468] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:28.139842 1203 tablet_bootstrap.cc:492] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:28.140012 1203 ts_tablet_manager.cc:1403] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.005s
I20251214 20:52:28.140520 991 leader_election.cc:304] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.140848 904 raft_consensus.cc:3060] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.140851 1203 raft_consensus.cc:359] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.141213 1203 raft_consensus.cc:385] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.141520 1203 raft_consensus.cc:740] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.142076 1203 consensus_queue.cc:260] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.142578 2561 raft_consensus.cc:2804] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:28.142668 2561 raft_consensus.cc:493] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:28.142750 2561 raft_consensus.cc:3060] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.143591 904 raft_consensus.cc:2468] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:28.144002 991 leader_election.cc:304] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.144397 2551 raft_consensus.cc:2804] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:28.144492 2551 raft_consensus.cc:697] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:28.145027 2551 consensus_queue.cc:237] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.145618 2561 raft_consensus.cc:515] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.146652 2561 leader_election.cc:290] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.138273 1202 tablet_bootstrap.cc:492] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:28.147300 1202 ts_tablet_manager.cc:1403] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.137s user 0.004s sys 0.000s
I20251214 20:52:28.149075 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58ac06f852fc4abea5cb213926002ca1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:28.151278 1202 raft_consensus.cc:359] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.151652 1202 raft_consensus.cc:385] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.151772 1202 raft_consensus.cc:740] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.152120 1202 consensus_queue.cc:260] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:28.154768 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0d0e8f35bf44c9a8797b7507dcce3b9 consensus state has no leader
I20251214 20:52:28.156126 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 69f6108792fa4c2a9744e3248c76251f (table default.Table_b3f5ab40d9334de1a194aab3347387d7 [id=b470f1b24a6d4e7ab4420ae212e54ff0]) successfully deleted
I20251214 20:52:28.158215 447 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e90d11d69b8d4fd49150aa67a8190e5b (table default.Table_c5bdb81886bf4361af53fd1a3b3b0b16 [id=b3df2580f4a74189aa06f021d6b4ce71]). Expected version 1 got 0
I20251214 20:52:28.158615 446 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:28.158798 1856 tablet_replica.cc:333] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:28.159439 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_bf1d2eb618b64a4e86cd121824fec0cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.160197 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf1d2eb618b64a4e86cd121824fec0cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.160122 1203 ts_tablet_manager.cc:1434] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20251214 20:52:28.160621 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 453cf734f8b34b59aa246d618ffeade5
I20251214 20:52:28.160962 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet d13f93bb5e5f4b59bcf9f4bfed070f25
I20251214 20:52:28.161257 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5bdf1b4c267c425ca1678f5de402fc9f" } modify_external_catalogs: true
I20251214 20:52:28.161459 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5bdf1b4c267c425ca1678f5de402fc9f" } modify_external_catalogs: true
I20251214 20:52:28.161558 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_dc35e103390944548c36be9ba4900882" } new_table_name: "default.Table_80d7626996ca40979563d484ae097f08" modify_external_catalogs: true
I20251214 20:52:28.161085 1856 raft_consensus.cc:2243] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.163089 448 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet a0976ffcc649461fac74e20b5e402275
I20251214 20:52:28.163606 448 catalog_manager.cc:6724] ReplaceTablet: tablet a0976ffcc649461fac74e20b5e402275 deleted and replaced by tablet 526f28f9aaad445081d9d69cc20ed7b3
I20251214 20:52:28.164063 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ee3c57e398fd468fa1e4e7cc22f6747a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.164737 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee3c57e398fd468fa1e4e7cc22f6747a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.165261 446 master_service.cc:954] ReplaceTablet: received request to replace tablet 956e4b4493f24109ab1251dde0a4cfc0 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:28.164077 1856 raft_consensus.cc:2272] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.165323 1203 tablet_bootstrap.cc:492] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:28.166940 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet a0976ffcc649461fac74e20b5e402275 with delete_type TABLET_DATA_DELETED (replaced by tablet 526f28f9aaad445081d9d69cc20ed7b3) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.161630 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_906242279a5b43858d131c378a5ea8da" } new_table_name: "default.Table_93730665b1ab4686a0d3e54740fbddc8" modify_external_catalogs: true
I20251214 20:52:28.168506 878 tablet_service.cc:1552] Processing DeleteTablet for tablet a0976ffcc649461fac74e20b5e402275 with delete_type TABLET_DATA_DELETED (replaced by tablet 526f28f9aaad445081d9d69cc20ed7b3) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.168634 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.168789 885 tablet_service.cc:1552] Processing DeleteTablet for tablet ebb26a5ae7224e9ca8a513c7b5013618 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.169199 877 tablet_service.cc:1552] Processing DeleteTablet for tablet 453cf734f8b34b59aa246d618ffeade5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.169445 890 tablet_service.cc:1552] Processing DeleteTablet for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.169600 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 58ac06f852fc4abea5cb213926002ca1 with delete_type TABLET_DATA_DELETED (replaced by tablet 248e3666fd13434e97378eb5a98dfadd) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.173743 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:28.173983 452 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:28.174633 1856 ts_tablet_manager.cc:1916] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.175539 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3a068835084e406eb4047a827faeb5d7" } modify_external_catalogs: true
I20251214 20:52:28.175776 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3a068835084e406eb4047a827faeb5d7" } modify_external_catalogs: true
I20251214 20:52:28.177016 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.182463 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 453cf734f8b34b59aa246d618ffeade5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.185200 1202 ts_tablet_manager.cc:1434] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20251214 20:52:28.186800 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 0a2fbec0543445b7856ef22c934c6840
I20251214 20:52:28.187378 449 catalog_manager.cc:6724] ReplaceTablet: tablet 0a2fbec0543445b7856ef22c934c6840 deleted and replaced by tablet 700ed62ef2e84719a7a8e9879feaf45c
I20251214 20:52:28.189349 1202 tablet_bootstrap.cc:492] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:28.190275 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a2fbec0543445b7856ef22c934c6840 with delete_type TABLET_DATA_DELETED (replaced by tablet 700ed62ef2e84719a7a8e9879feaf45c) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.190145 1203 tablet_bootstrap.cc:654] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.191087 1202 tablet_bootstrap.cc:654] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.193114 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a2fbec0543445b7856ef22c934c6840 with delete_type TABLET_DATA_DELETED (replaced by tablet 700ed62ef2e84719a7a8e9879feaf45c) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.193322 1982 rpcz_store.cc:276] 1214 20:52:26.825061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.826025 (+ 964us) service_pool.cc:225] Handling call
1214 20:52:28.041967 (+1215942us) ts_tablet_manager.cc:1151] Deleting tablet ca263a80702f4f64aa7c872c2be08207
1214 20:52:28.041976 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.050688 (+ 8712us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.053444 (+ 2756us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.138468 (+ 85024us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1215803}
I20251214 20:52:28.193877 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 372) took 1368 ms. Trace:
I20251214 20:52:28.194005 1982 rpcz_store.cc:276] 1214 20:52:26.825344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.827102 (+ 1758us) service_pool.cc:225] Handling call
1214 20:52:28.193726 (+1366624us) ts_tablet_manager.cc:1151] Deleting tablet 98105bceda9b44bf893f82a4234a3c81
1214 20:52:28.193738 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.193858 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1366440}
I20251214 20:52:28.194391 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 373) took 1178 ms. Trace:
I20251214 20:52:28.194510 1982 rpcz_store.cc:276] 1214 20:52:27.016008 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.016073 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:28.194274 (+1178201us) ts_tablet_manager.cc:1151] Deleting tablet 492770a7b8b74ba49315f974cc54eedb
1214 20:52:28.194283 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.194373 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1178001}
I20251214 20:52:28.205767 449 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet e90d11d69b8d4fd49150aa67a8190e5b (table default.Table_c5bdb81886bf4361af53fd1a3b3b0b16 [id=b3df2580f4a74189aa06f021d6b4ce71]). Expected version 1 got 0
I20251214 20:52:28.211577 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 374) took 1076 ms. Trace:
I20251214 20:52:28.211741 1982 rpcz_store.cc:276] 1214 20:52:27.135273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.135354 (+ 81us) service_pool.cc:225] Handling call
1214 20:52:28.211433 (+1076079us) ts_tablet_manager.cc:1151] Deleting tablet 6a000951f61948ac85fdd431f3d162c2
1214 20:52:28.211445 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.211559 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075762}
I20251214 20:52:28.212080 1982 tablet_replica.cc:333] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:28.212320 1982 raft_consensus.cc:2243] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.209985 1206 tablet_bootstrap.cc:492] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:28.212558 1206 ts_tablet_manager.cc:1403] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.001s
I20251214 20:52:28.213330 1206 raft_consensus.cc:359] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.213557 1206 raft_consensus.cc:385] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.213652 1206 raft_consensus.cc:740] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.213932 1206 consensus_queue.cc:260] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.215368 1977 ts_tablet_manager.cc:1916] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.219148 1206 ts_tablet_manager.cc:1434] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:28.220963 2470 raft_consensus.cc:493] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.221096 2470 raft_consensus.cc:515] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.221716 2470 leader_election.cc:290] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.222396 1206 tablet_bootstrap.cc:492] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:28.223246 712 leader_election.cc:343] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 2accf8a160584c82bb695a52e3f2b05c
I20251214 20:52:28.224254 1206 tablet_bootstrap.cc:654] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.229329 1202 tablet_bootstrap.cc:492] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:28.229550 1202 ts_tablet_manager.cc:1403] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20251214 20:52:28.230238 1203 tablet_bootstrap.cc:492] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:28.230468 1203 ts_tablet_manager.cc:1403] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.001s
I20251214 20:52:28.230376 1202 raft_consensus.cc:359] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.230562 1202 raft_consensus.cc:385] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.230690 1202 raft_consensus.cc:740] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.230964 1202 consensus_queue.cc:260] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.231490 1203 raft_consensus.cc:359] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.231715 1203 raft_consensus.cc:385] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.231797 1203 raft_consensus.cc:740] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.232100 1203 consensus_queue.cc:260] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.232559 1203 ts_tablet_manager.cc:1434] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:28.235152 1203 tablet_bootstrap.cc:492] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:28.239881 1203 tablet_bootstrap.cc:654] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.240257 1202 ts_tablet_manager.cc:1434] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20251214 20:52:28.241150 769 raft_consensus.cc:3060] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.244737 2572 raft_consensus.cc:493] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.244930 2572 raft_consensus.cc:515] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.245292 1202 tablet_bootstrap.cc:492] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:28.245674 2572 leader_election.cc:290] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:28.247009 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37a33cc9bb784f72956ac9f5493a0c3d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:28.248440 848 leader_election.cc:343] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:28.249583 851 leader_election.cc:343] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 37a33cc9bb784f72956ac9f5493a0c3d
I20251214 20:52:28.249701 851 leader_election.cc:304] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:28.249836 1202 tablet_bootstrap.cc:654] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.250313 2572 raft_consensus.cc:2749] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:28.252456 454 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:28.252799 454 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e90d11d69b8d4fd49150aa67a8190e5b (table default.Table_c5bdb81886bf4361af53fd1a3b3b0b16 [id=b3df2580f4a74189aa06f021d6b4ce71]). Expected version 1 got 0
I20251214 20:52:28.253115 454 catalog_manager.cc:5654] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:28.253535 454 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]). Expected version 1 got 0
I20251214 20:52:28.255976 1977 ts_tablet_manager.cc:1929] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251214 20:52:28.256294 1977 log.cc:1199] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/a142157acbcd44adb6099b4428103ee5
I20251214 20:52:28.256680 1977 ts_tablet_manager.cc:1950] T a142157acbcd44adb6099b4428103ee5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:28.258571 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 361) took 2355 ms. Trace:
I20251214 20:52:28.258703 1977 rpcz_store.cc:276] 1214 20:52:25.902937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.903001 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:28.107827 (+2204826us) ts_tablet_manager.cc:1151] Deleting tablet a142157acbcd44adb6099b4428103ee5
1214 20:52:28.107837 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.218322 (+110485us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.255859 (+ 37537us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.258551 (+ 2692us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2204465}
I20251214 20:52:28.259259 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 362) took 2314 ms. Trace:
I20251214 20:52:28.259356 1977 rpcz_store.cc:276] 1214 20:52:25.944845 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.944907 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:28.259117 (+2314210us) ts_tablet_manager.cc:1151] Deleting tablet 1eff8a23d48c43e2bdbfb34f4bab6edf
1214 20:52:28.259149 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.259245 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2313966}
W20251214 20:52:28.259471 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8db4364b5023438bb43538855007c95e consensus state has no leader
I20251214 20:52:28.260282 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 364) took 2315 ms. Trace:
I20251214 20:52:28.260326 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet a142157acbcd44adb6099b4428103ee5 (table default.Table_6a7108611108447db74a6feddaeea2b5 [id=0a848dd5f6474712b2eba70add3743dc]) successfully deleted
I20251214 20:52:28.260438 1977 rpcz_store.cc:276] 1214 20:52:25.945091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.945194 (+ 103us) service_pool.cc:225] Handling call
1214 20:52:28.260177 (+2314983us) ts_tablet_manager.cc:1151] Deleting tablet c9488c3505f5443998531cf749722be9
1214 20:52:28.260188 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.260267 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2314388}
W20251214 20:52:28.260587 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1eff8a23d48c43e2bdbfb34f4bab6edf already in progress: creating tablet
W20251214 20:52:28.261149 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c9488c3505f5443998531cf749722be9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9488c3505f5443998531cf749722be9 already in progress: creating tablet
I20251214 20:52:28.261487 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 363) took 2316 ms. Trace:
I20251214 20:52:28.261606 1977 rpcz_store.cc:276] 1214 20:52:25.944988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:25.945141 (+ 153us) service_pool.cc:225] Handling call
1214 20:52:28.261389 (+2316248us) ts_tablet_manager.cc:1151] Deleting tablet 2f97e5bad46343098418e8e34334b892
1214 20:52:28.261398 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.261471 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2315989}
W20251214 20:52:28.262028 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2f97e5bad46343098418e8e34334b892 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f97e5bad46343098418e8e34334b892 already in progress: creating tablet
I20251214 20:52:28.262565 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_0e2b5f246065408b923f196c464eceb7" } new_table_name: "default.Table_6085c994e5404569b349fe215f389970" modify_external_catalogs: true
I20251214 20:52:28.263259 1202 tablet_bootstrap.cc:492] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:28.263379 1202 ts_tablet_manager.cc:1403] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20251214 20:52:28.264089 1202 raft_consensus.cc:359] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.264295 1202 raft_consensus.cc:385] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.264393 1202 raft_consensus.cc:740] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.264674 1202 consensus_queue.cc:260] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.268141 1202 ts_tablet_manager.cc:1434] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:28.268483 1202 tablet_bootstrap.cc:492] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:28.268934 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 088c72a6f5014b6fbc6ee060b6b7a590
I20251214 20:52:28.271154 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 088c72a6f5014b6fbc6ee060b6b7a590 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.271767 1856 ts_tablet_manager.cc:1929] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:28.275306 1982 raft_consensus.cc:2272] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.275656 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 365) took 2297 ms. Trace:
I20251214 20:52:28.275750 1977 rpcz_store.cc:276] 1214 20:52:25.978448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.012230 (+ 33782us) service_pool.cc:225] Handling call
1214 20:52:28.275517 (+2263287us) ts_tablet_manager.cc:1151] Deleting tablet 57a6232564194cbba12182f3a29c2ebb
1214 20:52:28.275529 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.275639 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2263030}
I20251214 20:52:28.276088 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 366) took 1976 ms. Trace:
I20251214 20:52:28.276166 1977 rpcz_store.cc:276] 1214 20:52:26.299721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.299779 (+ 58us) service_pool.cc:225] Handling call
1214 20:52:28.275994 (+1976215us) ts_tablet_manager.cc:1151] Deleting tablet e59b80ee919b4e1393fe317ed0e3a3d2
1214 20:52:28.276002 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.276073 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1976025}
W20251214 20:52:28.276224 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 57a6232564194cbba12182f3a29c2ebb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57a6232564194cbba12182f3a29c2ebb already in progress: creating tablet
I20251214 20:52:28.276458 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 367) took 1957 ms. Trace:
I20251214 20:52:28.276533 1977 rpcz_store.cc:276] 1214 20:52:26.318600 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.318663 (+ 63us) service_pool.cc:225] Handling call
1214 20:52:28.276369 (+1957706us) ts_tablet_manager.cc:1151] Deleting tablet d745583012b4465090d85c75470e0bc2
1214 20:52:28.276377 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.276444 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1948152}
W20251214 20:52:28.276649 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet e59b80ee919b4e1393fe317ed0e3a3d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e59b80ee919b4e1393fe317ed0e3a3d2 already in progress: creating tablet
I20251214 20:52:28.276836 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 368) took 1912 ms. Trace:
I20251214 20:52:28.276916 1977 rpcz_store.cc:276] 1214 20:52:26.364856 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.367501 (+ 2645us) service_pool.cc:225] Handling call
1214 20:52:28.276749 (+1909248us) ts_tablet_manager.cc:1151] Deleting tablet e7b247b761c94c60ad3a468db371b30b
1214 20:52:28.276757 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.276822 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1908992}
W20251214 20:52:28.276993 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet d745583012b4465090d85c75470e0bc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d745583012b4465090d85c75470e0bc2 already in progress: creating tablet
I20251214 20:52:28.277230 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 369) took 1780 ms. Trace:
W20251214 20:52:28.277359 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet e7b247b761c94c60ad3a468db371b30b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7b247b761c94c60ad3a468db371b30b already in progress: creating tablet
I20251214 20:52:28.277312 1977 rpcz_store.cc:276] 1214 20:52:26.496358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.496421 (+ 63us) service_pool.cc:225] Handling call
1214 20:52:28.277144 (+1780723us) ts_tablet_manager.cc:1151] Deleting tablet f1eb4c3aa654480caaa69dbd5047365c
1214 20:52:28.277151 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.277216 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1780524}
I20251214 20:52:28.277599 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 376) took 1525 ms. Trace:
W20251214 20:52:28.277679 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet f1eb4c3aa654480caaa69dbd5047365c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1eb4c3aa654480caaa69dbd5047365c already in progress: creating tablet
I20251214 20:52:28.277681 1977 rpcz_store.cc:276] 1214 20:52:26.752541 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.758235 (+ 5694us) service_pool.cc:225] Handling call
1214 20:52:28.277501 (+1519266us) ts_tablet_manager.cc:1151] Deleting tablet 4f358df4d9414dcdb0d197f434e28118
1214 20:52:28.277508 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.277586 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519017}
I20251214 20:52:28.277923 1977 tablet_replica.cc:333] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
W20251214 20:52:28.278060 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 4f358df4d9414dcdb0d197f434e28118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f358df4d9414dcdb0d197f434e28118 already in progress: creating tablet
I20251214 20:52:28.278121 1977 raft_consensus.cc:2243] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.278230 1982 ts_tablet_manager.cc:1916] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.278306 1977 raft_consensus.cc:2272] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.279482 1202 tablet_bootstrap.cc:654] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.268790 962 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:28.270323 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 088c72a6f5014b6fbc6ee060b6b7a590 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:28.281164 439 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2807af0754914688bd7a090b1bc32eb1 (table default.Table_3b90ca08fe374acfa5c4ff6bf90cabb4 [id=9d1473df58ee486aa6a55893aca9d54c], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2807af0754914688bd7a090b1bc32eb1 consensus state has no leader
I20251214 20:52:28.288497 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ca263a80702f4f64aa7c872c2be08207 (table default.Table_642da83f7ed0444688afdf91b7a9e7e8 [id=bca3a2d37f8b4fc797a5c52a43c14869]) successfully deleted
W20251214 20:52:28.288923 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 98105bceda9b44bf893f82a4234a3c81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98105bceda9b44bf893f82a4234a3c81 already in progress: creating tablet
W20251214 20:52:28.289312 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 492770a7b8b74ba49315f974cc54eedb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 492770a7b8b74ba49315f974cc54eedb already in progress: creating tablet
W20251214 20:52:28.289680 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 6a000951f61948ac85fdd431f3d162c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a000951f61948ac85fdd431f3d162c2 already in progress: creating tablet
I20251214 20:52:28.290689 743 tablet_service.cc:1552] Processing DeleteTablet for tablet d13f93bb5e5f4b59bcf9f4bfed070f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.290766 749 tablet_service.cc:1552] Processing DeleteTablet for tablet a0976ffcc649461fac74e20b5e402275 with delete_type TABLET_DATA_DELETED (replaced by tablet 526f28f9aaad445081d9d69cc20ed7b3) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.290997 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a2fbec0543445b7856ef22c934c6840 with delete_type TABLET_DATA_DELETED (replaced by tablet 700ed62ef2e84719a7a8e9879feaf45c) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.291071 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 088c72a6f5014b6fbc6ee060b6b7a590 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:28.291249 713 leader_election.cc:343] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 2accf8a160584c82bb695a52e3f2b05c
I20251214 20:52:28.291383 713 leader_election.cc:304] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:28.291630 2470 raft_consensus.cc:2749] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:28.295825 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 453cf734f8b34b59aa246d618ffeade5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.296454 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbbfcef827ee4d548939aaf4783c8109" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:28.296658 774 raft_consensus.cc:2468] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:28.296800 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cbbfcef827ee4d548939aaf4783c8109" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:28.296999 775 raft_consensus.cc:3060] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.297399 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58ac06f852fc4abea5cb213926002ca1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:28.298377 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_a07fe706644d4c63b966f0751b757642"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:28.299283 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58ac06f852fc4abea5cb213926002ca1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:28.300079 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a07fe706644d4c63b966f0751b757642 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.300938 1206 tablet_bootstrap.cc:492] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:28.301056 1206 ts_tablet_manager.cc:1403] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.000s
I20251214 20:52:28.301646 775 raft_consensus.cc:2468] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:28.301739 1206 raft_consensus.cc:359] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.301931 1206 raft_consensus.cc:385] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.302001 1206 raft_consensus.cc:740] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
W20251214 20:52:28.302629 851 consensus_peers.cc:597] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec85857f047e44d78835305ddcb70768. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:28.302232 1206 consensus_queue.cc:260] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.303226 1206 ts_tablet_manager.cc:1434] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:28.303520 1206 tablet_bootstrap.cc:492] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:28.305428 1206 tablet_bootstrap.cc:654] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.309741 1206 tablet_bootstrap.cc:492] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:28.313328 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_68c79df4e9e0446f81c8500bcfa423a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.314172 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68c79df4e9e0446f81c8500bcfa423a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:28.313898 712 consensus_peers.cc:597] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2e83bca8ed642b6a1b705ef70fde27f. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:28.310305 1206 ts_tablet_manager.cc:1403] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20251214 20:52:28.315106 1206 raft_consensus.cc:359] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.315317 1206 raft_consensus.cc:385] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.315366 1206 raft_consensus.cc:740] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.317377 1206 consensus_queue.cc:260] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.320029 1206 ts_tablet_manager.cc:1434] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20251214 20:52:28.320727 1206 tablet_bootstrap.cc:492] T 00cf456eda464f5ba22d6893f73b5cba P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:28.321715 1977 ts_tablet_manager.cc:1916] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.322300 1206 tablet_bootstrap.cc:654] T 00cf456eda464f5ba22d6893f73b5cba P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.325013 2572 raft_consensus.cc:493] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.325268 2572 raft_consensus.cc:515] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.325982 2572 leader_election.cc:290] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:28.329937 851 leader_election.cc:343] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 69f6108792fa4c2a9744e3248c76251f
I20251214 20:52:28.330441 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69f6108792fa4c2a9744e3248c76251f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:28.331084 848 leader_election.cc:343] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.331230 848 leader_election.cc:304] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:28.331617 1206 tablet_bootstrap.cc:492] T 00cf456eda464f5ba22d6893f73b5cba P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:28.331733 2572 raft_consensus.cc:2749] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:28.332262 1206 ts_tablet_manager.cc:1403] T 00cf456eda464f5ba22d6893f73b5cba P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20251214 20:52:28.333202 1206 raft_consensus.cc:359] T 00cf456eda464f5ba22d6893f73b5cba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.334251 1206 raft_consensus.cc:385] T 00cf456eda464f5ba22d6893f73b5cba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.334419 1206 raft_consensus.cc:740] T 00cf456eda464f5ba22d6893f73b5cba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.334822 1206 consensus_queue.cc:260] T 00cf456eda464f5ba22d6893f73b5cba P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.335688 1206 ts_tablet_manager.cc:1434] T 00cf456eda464f5ba22d6893f73b5cba P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251214 20:52:28.335737 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ee7a207364d74e6b81d6a72e57cacbd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.336468 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee7a207364d74e6b81d6a72e57cacbd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.338349 1856 log.cc:1199] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/242a77ae57174de48f4eaf43384ee0ba
I20251214 20:52:28.339303 1856 ts_tablet_manager.cc:1950] T 242a77ae57174de48f4eaf43384ee0ba P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:28.340480 2364 consensus_queue.cc:1048] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:28.341259 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 428) took 1194 ms. Trace:
I20251214 20:52:28.341502 1856 rpcz_store.cc:276] 1214 20:52:27.146946 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.149964 (+ 3018us) service_pool.cc:225] Handling call
1214 20:52:28.158138 (+1008174us) ts_tablet_manager.cc:1151] Deleting tablet 242a77ae57174de48f4eaf43384ee0ba
1214 20:52:28.158155 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.213786 (+ 55631us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.271602 (+ 57816us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.341208 (+ 69606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007613}
I20251214 20:52:28.342423 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 242a77ae57174de48f4eaf43384ee0ba (table default.Table_9ad469dea994477983bfe62eef37e334 [id=b1be575d16354425ab4f7e8e62847820]) successfully deleted
W20251214 20:52:28.342716 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5d28f09ad174936b47debace0599600 has no consensus state
I20251214 20:52:28.343065 1856 tablet_replica.cc:333] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:28.343986 2470 raft_consensus.cc:493] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.344139 2470 raft_consensus.cc:515] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.344746 2470 leader_election.cc:290] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:28.346093 712 leader_election.cc:343] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: a38e82f5dd1442f3a1dfcd6fbd94dbaf
I20251214 20:52:28.349970 1206 tablet_bootstrap.cc:492] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:28.351378 989 leader_election.cc:343] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:28.351881 989 leader_election.cc:343] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:28.352237 989 consensus_peers.cc:597] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:28.343508 1856 raft_consensus.cc:2243] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.356053 904 raft_consensus.cc:3060] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.358026 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_9ff099dcfac14636a9dcaa70207c86a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:28.357802 1856 raft_consensus.cc:2272] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:28.358671 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ff099dcfac14636a9dcaa70207c86a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.360494 1206 tablet_bootstrap.cc:654] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.366541 446 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 956e4b4493f24109ab1251dde0a4cfc0
I20251214 20:52:28.367053 446 catalog_manager.cc:6724] ReplaceTablet: tablet 956e4b4493f24109ab1251dde0a4cfc0 deleted and replaced by tablet d39e245bc5a04c1186d0973645aa1b54
I20251214 20:52:28.367606 1202 tablet_bootstrap.cc:492] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:28.369215 1856 ts_tablet_manager.cc:1916] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.371944 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 956e4b4493f24109ab1251dde0a4cfc0 with delete_type TABLET_DATA_DELETED (replaced by tablet d39e245bc5a04c1186d0973645aa1b54) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.372740 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 956e4b4493f24109ab1251dde0a4cfc0 with delete_type TABLET_DATA_DELETED (replaced by tablet d39e245bc5a04c1186d0973645aa1b54) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.372097 1202 ts_tablet_manager.cc:1403] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.104s user 0.006s sys 0.015s
I20251214 20:52:28.384234 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 956e4b4493f24109ab1251dde0a4cfc0 with delete_type TABLET_DATA_DELETED (replaced by tablet d39e245bc5a04c1186d0973645aa1b54) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.385214 1202 raft_consensus.cc:359] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.386648 1982 ts_tablet_manager.cc:1929] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:28.386583 1202 raft_consensus.cc:385] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.386847 1982 log.cc:1199] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/012b81223d0f4ae9b742014e329f11a9
I20251214 20:52:28.387109 1202 raft_consensus.cc:740] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.387777 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a941092a7b2944bfa37073939660b587" } new_table_name: "default.Table_dce91ff3d9c74c7db1d400a24c4e550a" modify_external_catalogs: true
I20251214 20:52:28.389101 1202 consensus_queue.cc:260] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.390089 904 raft_consensus.cc:2468] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:28.391798 2364 raft_consensus.cc:493] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.391953 2364 raft_consensus.cc:515] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.392581 2364 leader_election.cc:290] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.393059 991 leader_election.cc:304] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:28.393707 2364 raft_consensus.cc:2804] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:28.393817 2364 raft_consensus.cc:697] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:28.394047 2364 consensus_queue.cc:237] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.394356 452 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_a426957d72cc493393637b00a3b1e12d [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde] alter complete (version 2)
I20251214 20:52:28.394338 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f2314d33fa474ca36dafe0a0eae5ae" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.395319 1202 ts_tablet_manager.cc:1434] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
I20251214 20:52:28.397435 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 69f6108792fa4c2a9744e3248c76251f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.396260 904 raft_consensus.cc:2468] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
W20251214 20:52:28.400445 713 leader_election.cc:343] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: a38e82f5dd1442f3a1dfcd6fbd94dbaf
I20251214 20:52:28.400627 713 leader_election.cc:304] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:28.400712 991 leader_election.cc:304] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.401006 2551 raft_consensus.cc:493] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.401136 2551 raft_consensus.cc:515] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.401360 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f2314d33fa474ca36dafe0a0eae5ae" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:28.401787 2551 leader_election.cc:290] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.401914 2551 raft_consensus.cc:2804] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:28.401968 2551 raft_consensus.cc:493] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:28.402022 2551 raft_consensus.cc:3060] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.402289 1202 tablet_bootstrap.cc:492] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:28.404455 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ad0accc4d4ef4dbc82e826110493c26a" } modify_external_catalogs: true
I20251214 20:52:28.404657 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ad0accc4d4ef4dbc82e826110493c26a" } modify_external_catalogs: true
I20251214 20:52:28.405098 2551 raft_consensus.cc:515] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.405171 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_8f5d31355346492989caa148a8f1440b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.405742 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f5d31355346492989caa148a8f1440b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.405767 2551 leader_election.cc:290] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:28.406399 989 leader_election.cc:343] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.407805 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b1a937150ce4ecfa210c8465d682496" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.407886 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f2314d33fa474ca36dafe0a0eae5ae" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:28.408376 915 raft_consensus.cc:3060] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.408229 904 raft_consensus.cc:2468] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:28.408983 991 leader_election.cc:304] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.409309 2551 raft_consensus.cc:2804] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:28.409389 2551 raft_consensus.cc:493] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:28.409464 2551 raft_consensus.cc:3060] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.410951 915 raft_consensus.cc:2468] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:28.411513 991 leader_election.cc:304] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.411808 2604 raft_consensus.cc:2804] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:28.411835 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet b70579ddf96f4499b4e4f235e8f69d67
I20251214 20:52:28.411955 2604 raft_consensus.cc:697] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:28.412209 2604 consensus_queue.cc:237] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.412989 2551 raft_consensus.cc:515] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.413421 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet b70579ddf96f4499b4e4f235e8f69d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.414059 880 tablet_service.cc:1552] Processing DeleteTablet for tablet b70579ddf96f4499b4e4f235e8f69d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.414561 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b1a937150ce4ecfa210c8465d682496" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:28.415012 2551 leader_election.cc:290] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:28.415153 989 leader_election.cc:343] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.415498 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b1a937150ce4ecfa210c8465d682496" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:28.415681 915 raft_consensus.cc:3060] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.417609 1203 tablet_bootstrap.cc:492] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:28.417776 1203 ts_tablet_manager.cc:1403] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.183s user 0.044s sys 0.025s
I20251214 20:52:28.418620 1203 raft_consensus.cc:359] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.418797 1203 raft_consensus.cc:385] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.418849 1203 raft_consensus.cc:740] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.419049 1203 consensus_queue.cc:260] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.419628 1203 ts_tablet_manager.cc:1434] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:28.419948 1203 tablet_bootstrap.cc:492] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:28.421880 1203 tablet_bootstrap.cc:654] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.422660 915 raft_consensus.cc:2468] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:28.423313 991 leader_election.cc:304] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:28.423862 2551 raft_consensus.cc:2804] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:28.424140 2551 raft_consensus.cc:697] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:28.424475 2551 consensus_queue.cc:237] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.425894 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dca369dc5f54a4ea1783e334c5670d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.429113 447 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:28.430950 749 tablet_service.cc:1552] Processing DeleteTablet for tablet ec85857f047e44d78835305ddcb70768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.430469 1202 tablet_bootstrap.cc:654] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.443207 1982 ts_tablet_manager.cc:1950] T 012b81223d0f4ae9b742014e329f11a9 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:28.443365 747 tablet_service.cc:1552] Processing DeleteTablet for tablet bf8643c7e4094f529cd55656c7b3edb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.443225 2470 raft_consensus.cc:2749] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:28.443243 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b1a937150ce4ecfa210c8465d682496" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:28.443828 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 375) took 1289 ms. Trace:
I20251214 20:52:28.443950 1982 rpcz_store.cc:276] 1214 20:52:27.154640 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.154713 (+ 73us) service_pool.cc:225] Handling call
1214 20:52:28.212036 (+1057323us) ts_tablet_manager.cc:1151] Deleting tablet 012b81223d0f4ae9b742014e329f11a9
1214 20:52:28.212045 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.286239 (+ 74194us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.386539 (+100300us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.443810 (+ 57271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1057104}
I20251214 20:52:28.443259 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f2314d33fa474ca36dafe0a0eae5ae" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:28.443276 746 tablet_service.cc:1552] Processing DeleteTablet for tablet b70579ddf96f4499b4e4f235e8f69d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.444501 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 376) took 1196 ms. Trace:
W20251214 20:52:28.444466 989 leader_election.cc:343] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.444628 1982 rpcz_store.cc:276] 1214 20:52:27.248099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.248163 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:28.444390 (+1196227us) ts_tablet_manager.cc:1151] Deleting tablet 79e621f29c01448399d22615284fccd4
1214 20:52:28.444401 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.444486 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195960}
I20251214 20:52:28.444934 1982 tablet_replica.cc:333] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
W20251214 20:52:28.445216 989 leader_election.cc:343] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.445226 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 012b81223d0f4ae9b742014e329f11a9 (table default.Table_b151b87a38944a7a89d0fe6ff10ae68b [id=7b72926c96404fa5b84cfb86e6ae10a7]) successfully deleted
W20251214 20:52:28.445544 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 79e621f29c01448399d22615284fccd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79e621f29c01448399d22615284fccd4 already in progress: creating tablet
I20251214 20:52:28.446130 1982 raft_consensus.cc:2243] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.446363 1982 raft_consensus.cc:2272] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.447288 1856 ts_tablet_manager.cc:1929] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:28.448032 1856 log.cc:1199] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/0c973e0111804b8b869068bdcc5869bf
W20251214 20:52:28.449084 713 consensus_peers.cc:597] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 242a77ae57174de48f4eaf43384ee0ba. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:28.450469 1977 ts_tablet_manager.cc:1929] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:28.450652 1977 log.cc:1199] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ca57963bcfb54eec9fcbed82757d8387
I20251214 20:52:28.451005 1977 ts_tablet_manager.cc:1950] T ca57963bcfb54eec9fcbed82757d8387 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:28.450536 1856 ts_tablet_manager.cc:1950] T 0c973e0111804b8b869068bdcc5869bf P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:28.451876 1202 tablet_bootstrap.cc:492] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:28.452128 1202 ts_tablet_manager.cc:1403] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.001s
I20251214 20:52:28.453085 1202 raft_consensus.cc:359] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.453377 1202 raft_consensus.cc:385] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.453523 1202 raft_consensus.cc:740] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.453953 1202 consensus_queue.cc:260] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.455050 1202 ts_tablet_manager.cc:1434] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:28.457931 1202 tablet_bootstrap.cc:492] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:28.458079 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 432) took 1268 ms. Trace:
I20251214 20:52:28.458940 1856 rpcz_store.cc:276] 1214 20:52:27.189536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.189676 (+ 140us) service_pool.cc:225] Handling call
1214 20:52:28.342944 (+1153268us) ts_tablet_manager.cc:1151] Deleting tablet 0c973e0111804b8b869068bdcc5869bf
1214 20:52:28.342983 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.382594 (+ 39611us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.446140 (+ 63546us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.458052 (+ 11912us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1153060}
I20251214 20:52:28.461439 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 0c973e0111804b8b869068bdcc5869bf (table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f]) successfully deleted
I20251214 20:52:28.465903 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_2f947b9f382942e082ea4909657a8f34" } new_table_name: "default.Table_407ddd842d284ecd87a391f1b54a2b5a" modify_external_catalogs: true
I20251214 20:52:28.467571 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_f9499a2114f44353aaf65b35ed56f119"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.468232 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9499a2114f44353aaf65b35ed56f119 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.468804 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 434) took 1223 ms. Trace:
I20251214 20:52:28.469851 1856 rpcz_store.cc:276] 1214 20:52:27.245439 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.245531 (+ 92us) service_pool.cc:225] Handling call
1214 20:52:28.462470 (+1216939us) ts_tablet_manager.cc:1151] Deleting tablet 79e621f29c01448399d22615284fccd4
1214 20:52:28.464859 (+ 2389us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.468549 (+ 3690us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211144}
W20251214 20:52:28.472355 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 79e621f29c01448399d22615284fccd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79e621f29c01448399d22615284fccd4 already in progress: creating tablet
W20251214 20:52:28.474093 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1def5f0de5ce4eb0bed36f626c4f0683 consensus state has no leader
I20251214 20:52:28.477217 1856 tablet_replica.cc:333] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:28.482695 848 consensus_peers.cc:597] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:28.482565 1856 raft_consensus.cc:2243] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:28.485872 1202 tablet_bootstrap.cc:654] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.487069 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.486896 1856 raft_consensus.cc:2272] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.487795 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.492568 1856 ts_tablet_manager.cc:1916] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.502820 2571 consensus_queue.cc:1048] T 42bdcecba5534294aca1eb419f18e9d2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:28.503203 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_bf73b75ea64f496da60f3a9a6afa5d6e" } modify_external_catalogs: true
I20251214 20:52:28.503486 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_bf73b75ea64f496da60f3a9a6afa5d6e" } modify_external_catalogs: true
I20251214 20:52:28.509581 1982 ts_tablet_manager.cc:1916] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.513192 1202 tablet_bootstrap.cc:492] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:28.513504 1202 ts_tablet_manager.cc:1403] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.001s
I20251214 20:52:28.514539 1202 raft_consensus.cc:359] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.515322 1202 raft_consensus.cc:385] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.515476 1202 raft_consensus.cc:740] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.515908 1202 consensus_queue.cc:260] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.519559 1202 ts_tablet_manager.cc:1434] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:28.522146 1202 tablet_bootstrap.cc:492] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:28.523818 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 379) took 1680 ms. Trace:
I20251214 20:52:28.524003 1977 rpcz_store.cc:276] 1214 20:52:26.843783 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.843880 (+ 97us) service_pool.cc:225] Handling call
1214 20:52:28.277882 (+1434002us) ts_tablet_manager.cc:1151] Deleting tablet ca57963bcfb54eec9fcbed82757d8387
1214 20:52:28.277889 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.324893 (+ 47004us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.450342 (+125449us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.523788 (+ 73446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1433750}
I20251214 20:52:28.524528 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ca57963bcfb54eec9fcbed82757d8387 (table default.Table_0cc13ed7f70c4cabb5fdfb277cb7b9a0 [id=f7d7826ca95442bcac3a73bb3b171956]) successfully deleted
I20251214 20:52:28.524973 1977 tablet_replica.cc:333] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:28.525209 1977 raft_consensus.cc:2243] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.525418 1977 raft_consensus.cc:2272] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.526026 1977 ts_tablet_manager.cc:1916] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.535354 1206 tablet_bootstrap.cc:492] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:28.537011 448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_95769572161249d98020c723ac65b78e" } modify_external_catalogs: true
I20251214 20:52:28.537240 448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_95769572161249d98020c723ac65b78e" } modify_external_catalogs: true
I20251214 20:52:28.542188 1202 tablet_bootstrap.cc:654] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.544518 1206 ts_tablet_manager.cc:1403] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.195s user 0.001s sys 0.005s
I20251214 20:52:28.547341 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_8f475bc0f2dd4602b5f8e057241c3c16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.547977 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f475bc0f2dd4602b5f8e057241c3c16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.548108 1206 raft_consensus.cc:359] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.548264 1206 raft_consensus.cc:385] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.548307 1206 raft_consensus.cc:740] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.549675 1203 tablet_bootstrap.cc:492] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:28.549912 1203 ts_tablet_manager.cc:1403] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.130s user 0.020s sys 0.039s
I20251214 20:52:28.550825 1203 raft_consensus.cc:359] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.551038 1203 raft_consensus.cc:385] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.551147 1203 raft_consensus.cc:740] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.551429 1203 consensus_queue.cc:260] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.551976 1203 ts_tablet_manager.cc:1434] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:28.552412 1203 tablet_bootstrap.cc:492] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:28.553610 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4501fd5566a34e2a82009cc9ce6bdd37 consensus state has no leader
I20251214 20:52:28.553921 446 master_service.cc:954] ReplaceTablet: received request to replace tablet e6ebfc373fbf41b1920c271a15302484 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:28.558287 775 raft_consensus.cc:3060] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.561517 2572 consensus_queue.cc:1048] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:28.563578 1982 ts_tablet_manager.cc:1929] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:28.563719 1203 tablet_bootstrap.cc:654] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.564754 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_5cda3bedf2d24fc2b279c049146aa2c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.565487 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cda3bedf2d24fc2b279c049146aa2c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.569541 1203 tablet_bootstrap.cc:492] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:28.569733 1203 ts_tablet_manager.cc:1403] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20251214 20:52:28.570741 1203 raft_consensus.cc:359] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.570950 1203 raft_consensus.cc:385] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.571028 1203 raft_consensus.cc:740] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.571331 1203 consensus_queue.cc:260] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.572000 1203 ts_tablet_manager.cc:1434] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:28.572312 1203 tablet_bootstrap.cc:492] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:28.572175 1856 ts_tablet_manager.cc:1929] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:28.572889 1856 log.cc:1199] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/1e31615d1277412eb8bd80451f2c4366
I20251214 20:52:28.574361 1203 tablet_bootstrap.cc:654] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.583231 1202 tablet_bootstrap.cc:492] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:28.583586 1202 ts_tablet_manager.cc:1403] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.000s
I20251214 20:52:28.584544 1202 raft_consensus.cc:359] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.584844 1202 raft_consensus.cc:385] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.584980 1202 raft_consensus.cc:740] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.585410 1202 consensus_queue.cc:260] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.589337 452 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_31645b7a21d345ef9bc06c0ba9b2c5fa" } new_table_name: "default.Table_7306395aef6240c39e095e0c8815130b" modify_external_catalogs: true
I20251214 20:52:28.592119 1202 ts_tablet_manager.cc:1434] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251214 20:52:28.592518 1856 ts_tablet_manager.cc:1950] T 1e31615d1277412eb8bd80451f2c4366 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:28.598300 2364 raft_consensus.cc:493] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.598464 2364 raft_consensus.cc:515] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.599063 2364 leader_election.cc:290] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.609284 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e90d11d69b8d4fd49150aa67a8190e5b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.609262 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6a7108611108447db74a6feddaeea2b5" } modify_external_catalogs: true
I20251214 20:52:28.609462 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6a7108611108447db74a6feddaeea2b5" } modify_external_catalogs: true
I20251214 20:52:28.609797 910 raft_consensus.cc:2468] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:28.610368 991 leader_election.cc:304] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.610697 2364 raft_consensus.cc:2804] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:28.610771 2364 raft_consensus.cc:493] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:28.610832 2364 raft_consensus.cc:3060] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.613416 2364 raft_consensus.cc:515] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.616344 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e90d11d69b8d4fd49150aa67a8190e5b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:28.616636 2364 leader_election.cc:290] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.617154 910 raft_consensus.cc:3060] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.617527 1202 tablet_bootstrap.cc:492] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:28.619911 910 raft_consensus.cc:2468] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:28.622992 991 leader_election.cc:304] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.623378 2364 raft_consensus.cc:2804] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:28.623577 2364 raft_consensus.cc:697] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:28.623859 2364 consensus_queue.cc:237] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.628096 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 437) took 1186 ms. Trace:
I20251214 20:52:28.628417 2551 raft_consensus.cc:493] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.628563 2551 raft_consensus.cc:515] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.628366 1856 rpcz_store.cc:276] 1214 20:52:27.441795 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.441941 (+ 146us) service_pool.cc:225] Handling call
1214 20:52:28.476669 (+1034728us) ts_tablet_manager.cc:1151] Deleting tablet 1e31615d1277412eb8bd80451f2c4366
1214 20:52:28.476691 (+ 22us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.539051 (+ 62360us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.564518 (+ 25467us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.628056 (+ 63538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020894}
I20251214 20:52:28.629194 2551 leader_election.cc:290] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.629225 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]) successfully deleted
I20251214 20:52:28.630210 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "876d2a1f1d1c4f6882a5857b43568ee5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:28.630703 991 leader_election.cc:343] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.631459 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 438) took 1189 ms. Trace:
I20251214 20:52:28.631678 1856 rpcz_store.cc:276] 1214 20:52:27.441882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.442088 (+ 206us) service_pool.cc:225] Handling call
1214 20:52:28.630294 (+1188206us) ts_tablet_manager.cc:1151] Deleting tablet cea9423c40044e4cabb8bbd755a46688
1214 20:52:28.631102 (+ 808us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.631413 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1174206}
W20251214 20:52:28.633033 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet cea9423c40044e4cabb8bbd755a46688 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cea9423c40044e4cabb8bbd755a46688 already in progress: creating tablet
I20251214 20:52:28.634538 1856 tablet_replica.cc:333] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:28.635219 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_0f409000b9514206a08757040e81a6a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.635968 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f409000b9514206a08757040e81a6a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.636924 1856 raft_consensus.cc:2243] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:28.637687 1202 tablet_bootstrap.cc:654] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.566488 1206 consensus_queue.cc:260] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.643733 1206 ts_tablet_manager.cc:1434] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.096s user 0.000s sys 0.001s
I20251214 20:52:28.644042 1206 tablet_bootstrap.cc:492] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:28.645780 1206 tablet_bootstrap.cc:654] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.602658 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e90d11d69b8d4fd49150aa67a8190e5b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:28.615350 2635 tablet.cc:1722] T 8d72f55d2d2440d993aa205d96c356f1 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:28.615458 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e90d11d69b8d4fd49150aa67a8190e5b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:28.630074 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "876d2a1f1d1c4f6882a5857b43568ee5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:28.647033 989 leader_election.cc:343] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:28.647706 989 leader_election.cc:343] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:28.647994 989 leader_election.cc:343] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.648085 989 leader_election.cc:304] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:28.648712 2551 raft_consensus.cc:2749] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:28.651547 1856 raft_consensus.cc:2272] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.651928 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:28.657593 2551 raft_consensus.cc:493] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.657794 2551 raft_consensus.cc:515] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:28.658031 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 58940us
I20251214 20:52:28.658540 2551 leader_election.cc:290] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.662424 1982 log.cc:1199] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/4aa1a44452074ec28a1bd9c48a5f47cb
I20251214 20:52:28.662907 1982 ts_tablet_manager.cc:1950] T 4aa1a44452074ec28a1bd9c48a5f47cb P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:28.663276 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 377) took 1227 ms. Trace:
I20251214 20:52:28.663371 1982 rpcz_store.cc:276] 1214 20:52:27.436149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.436481 (+ 332us) service_pool.cc:225] Handling call
1214 20:52:28.444892 (+1008411us) ts_tablet_manager.cc:1151] Deleting tablet 4aa1a44452074ec28a1bd9c48a5f47cb
1214 20:52:28.444900 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.557615 (+112715us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.563470 (+ 5855us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.663260 (+ 99790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008209}
I20251214 20:52:28.663713 1982 tablet_replica.cc:333] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:28.663944 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 4aa1a44452074ec28a1bd9c48a5f47cb (table default.Table_dbcc1409fc7e48cb804842a37eebaad2 [id=81c0278b4add47d9947b461866fe9eae]) successfully deleted
I20251214 20:52:28.665148 1856 ts_tablet_manager.cc:1916] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.666460 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_f7d781182d07441a824d887e07286dc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.667065 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7d781182d07441a824d887e07286dc3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.672415 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98b7cac3930a4a9eb16ac413c8f1e092" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.672705 910 raft_consensus.cc:2468] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:28.673197 991 leader_election.cc:304] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.673718 2551 raft_consensus.cc:2804] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:28.673806 2551 raft_consensus.cc:493] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:28.673867 2551 raft_consensus.cc:3060] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.673878 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98b7cac3930a4a9eb16ac413c8f1e092" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:28.674355 989 leader_election.cc:343] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.677356 2551 raft_consensus.cc:515] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:28.678738 851 consensus_peers.cc:597] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c973e0111804b8b869068bdcc5869bf. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:28.679225 2551 leader_election.cc:290] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.679392 2634 tablet.cc:1722] T 8d72f55d2d2440d993aa205d96c356f1 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:28.679745 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98b7cac3930a4a9eb16ac413c8f1e092" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:28.680032 910 raft_consensus.cc:3060] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.682389 1206 tablet_bootstrap.cc:492] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:28.691092 452 master_service.cc:954] ReplaceTablet: received request to replace tablet ed9790fbebda4f848291930bf1dd8174 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:28.692385 2439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=423 overwritten=0 applied=423 ignored=0} inserts{seen=402 ignored=0} mutations{seen=475 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251214 20:52:28.693337 2439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Bootstrap complete.
I20251214 20:52:28.697022 2364 raft_consensus.cc:493] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.697700 2364 raft_consensus.cc:515] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.698371 2364 leader_election.cc:290] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.698908 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47e813373fbd482298312ead4f037c01" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.699262 904 raft_consensus.cc:2468] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:28.699308 1206 ts_tablet_manager.cc:1403] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.055s user 0.000s sys 0.004s
I20251214 20:52:28.699746 991 leader_election.cc:304] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.685366 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98b7cac3930a4a9eb16ac413c8f1e092" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:28.688006 713 consensus_peers.cc:597] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2e83bca8ed642b6a1b705ef70fde27f. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:28.700105 1206 raft_consensus.cc:359] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.700385 1206 raft_consensus.cc:385] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.700438 1206 raft_consensus.cc:740] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.700629 1206 consensus_queue.cc:260] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.700802 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47e813373fbd482298312ead4f037c01" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:28.700992 1206 ts_tablet_manager.cc:1434] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:28.701323 1206 tablet_bootstrap.cc:492] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:28.702281 989 leader_election.cc:343] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.702941 1206 tablet_bootstrap.cc:654] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.704706 452 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8e83dc5fa8e547f6bfdc4ddba95fcb78" } new_table_name: "default.Table_adb418bf04534b868c044988fb97e906" modify_external_catalogs: true
W20251214 20:52:28.707088 989 leader_election.cc:343] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.710925 2439 raft_consensus.cc:359] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:28.712260 2439 raft_consensus.cc:740] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6713a20a82b434f8a8bc122113e26ad, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.713402 2439 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 423, Last appended: 3.423, Last appended by leader: 423, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:28.715711 447 master_service.cc:954] ReplaceTablet: received request to replace tablet 6b389a63a58d438db41a343db6e04a88 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:28.718439 1203 tablet_bootstrap.cc:492] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:28.718603 1203 ts_tablet_manager.cc:1403] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.146s user 0.023s sys 0.029s
I20251214 20:52:28.719395 1203 raft_consensus.cc:359] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.719575 1203 raft_consensus.cc:385] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.719635 1203 raft_consensus.cc:740] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.719827 1203 consensus_queue.cc:260] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.720346 1203 ts_tablet_manager.cc:1434] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:28.720688 1203 tablet_bootstrap.cc:492] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:28.722759 1203 tablet_bootstrap.cc:654] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:28.723618 712 consensus_peers.cc:597] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 242a77ae57174de48f4eaf43384ee0ba. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:28.723781 2551 raft_consensus.cc:2804] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:28.723868 2551 raft_consensus.cc:493] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:28.723928 2551 raft_consensus.cc:3060] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.725481 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ae7fe3d1672c46baa454f28810e589af" } modify_external_catalogs: true
I20251214 20:52:28.725679 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ae7fe3d1672c46baa454f28810e589af" } modify_external_catalogs: true
I20251214 20:52:28.726534 2551 raft_consensus.cc:515] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.729215 2470 raft_consensus.cc:493] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.729477 2470 raft_consensus.cc:515] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.730278 2470 leader_election.cc:290] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.731230 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf8643c7e4094f529cd55656c7b3edb6" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.731498 903 raft_consensus.cc:2468] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
W20251214 20:52:28.733220 713 leader_election.cc:343] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: bf8643c7e4094f529cd55656c7b3edb6
I20251214 20:52:28.733253 446 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_407ddd842d284ecd87a391f1b54a2b5a [id=66d124c2a8dd4cc188a832eff1dbe9ed] alter complete (version 1)
I20251214 20:52:28.733484 2551 leader_election.cc:290] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.733718 712 leader_election.cc:304] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:28.734522 746 tablet_service.cc:1552] Processing DeleteTablet for tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 with delete_type TABLET_DATA_DELETED (replaced by tablet 15b3216f3d2b443e8ab2f994445956f9) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.734635 2470 raft_consensus.cc:2804] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:28.734721 2470 raft_consensus.cc:493] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:28.734367 1982 raft_consensus.cc:2243] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.734783 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47e813373fbd482298312ead4f037c01" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:28.734956 1982 raft_consensus.cc:2272] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.735523 1982 ts_tablet_manager.cc:1916] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:28.735543 989 leader_election.cc:343] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.734781 2470 raft_consensus.cc:3060] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.736879 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3b90ca08fe374acfa5c4ff6bf90cabb4" } new_table_name: "default.Table_c6c0496d455a4db68c05c0ff8b06493d" modify_external_catalogs: true
I20251214 20:52:28.738943 2470 raft_consensus.cc:515] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.740352 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47e813373fbd482298312ead4f037c01" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:28.740717 903 raft_consensus.cc:3060] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:28.742794 452 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96157aae81624a4ba9fcf8c3f142911f has no consensus state
W20251214 20:52:28.745466 713 leader_election.cc:343] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: bf8643c7e4094f529cd55656c7b3edb6
I20251214 20:52:28.746587 2470 leader_election.cc:290] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.746716 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf8643c7e4094f529cd55656c7b3edb6" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:28.746937 908 raft_consensus.cc:3060] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.748104 903 raft_consensus.cc:2468] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:28.748858 991 leader_election.cc:304] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:28.749531 908 raft_consensus.cc:2468] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:28.749926 2604 raft_consensus.cc:493] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.750097 1977 ts_tablet_manager.cc:1929] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:28.750247 1977 log.cc:1199] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/98105bceda9b44bf893f82a4234a3c81
I20251214 20:52:28.750454 2604 raft_consensus.cc:515] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.750639 1977 ts_tablet_manager.cc:1950] T 98105bceda9b44bf893f82a4234a3c81 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:28.750958 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 380) took 1907 ms. Trace:
I20251214 20:52:28.751077 1977 rpcz_store.cc:276] 1214 20:52:26.843906 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.844186 (+ 280us) service_pool.cc:225] Handling call
1214 20:52:28.524832 (+1680646us) ts_tablet_manager.cc:1151] Deleting tablet 98105bceda9b44bf893f82a4234a3c81
1214 20:52:28.524843 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.630651 (+105808us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.749994 (+119343us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.750944 (+ 950us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1680461}
I20251214 20:52:28.751507 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 381) took 1902 ms. Trace:
I20251214 20:52:28.751632 1977 rpcz_store.cc:276] 1214 20:52:26.849218 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.851038 (+ 1820us) service_pool.cc:225] Handling call
1214 20:52:28.751416 (+1900378us) ts_tablet_manager.cc:1151] Deleting tablet 5da9deeedce0421c9aeaebd71ba11b1b
1214 20:52:28.751424 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.751493 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900141}
I20251214 20:52:28.751933 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 382) took 1902 ms. Trace:
I20251214 20:52:28.752017 1977 rpcz_store.cc:276] 1214 20:52:26.849347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.851316 (+ 1969us) service_pool.cc:225] Handling call
1214 20:52:28.751857 (+1900541us) ts_tablet_manager.cc:1151] Deleting tablet 762068d560ff47918dd9fd8791083433
1214 20:52:28.751866 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.751918 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900409}
I20251214 20:52:28.752417 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 383) took 1903 ms. Trace:
I20251214 20:52:28.752475 2604 leader_election.cc:290] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.752467 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_b6011f3e00584819944d165d653d06ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:28.752875 2604 raft_consensus.cc:2804] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:28.752936 1977 rpcz_store.cc:276] 1214 20:52:26.849453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.851477 (+ 2024us) service_pool.cc:225] Handling call
1214 20:52:28.752349 (+1900872us) ts_tablet_manager.cc:1151] Deleting tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf
1214 20:52:28.752356 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.752404 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900723}
W20251214 20:52:28.753070 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6011f3e00584819944d165d653d06ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.753232 2604 raft_consensus.cc:697] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:28.753650 2604 consensus_queue.cc:237] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.756181 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]) successfully deleted
W20251214 20:52:28.756433 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 5da9deeedce0421c9aeaebd71ba11b1b because the tablet was not found. No further retry: Not found: Tablet not found: 5da9deeedce0421c9aeaebd71ba11b1b
W20251214 20:52:28.756662 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 762068d560ff47918dd9fd8791083433 because the tablet was not found. No further retry: Not found: Tablet not found: 762068d560ff47918dd9fd8791083433
W20251214 20:52:28.760962 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 65003.4us
I20251214 20:52:28.757010 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "453cf734f8b34b59aa246d618ffeade5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.761518 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 384) took 1911 ms. Trace:
W20251214 20:52:28.761502 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf because the tablet was not found. No further retry: Not found: Tablet not found: a38e82f5dd1442f3a1dfcd6fbd94dbaf
I20251214 20:52:28.761582 1977 rpcz_store.cc:276] 1214 20:52:26.849571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.863836 (+ 14265us) service_pool.cc:225] Handling call
1214 20:52:28.761439 (+1897603us) ts_tablet_manager.cc:1151] Deleting tablet 35a529702f924a5c9bf53c3c668c6094
1214 20:52:28.761450 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.761505 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1887067}
W20251214 20:52:28.761970 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 35a529702f924a5c9bf53c3c668c6094 because the tablet was not found. No further retry: Not found: Tablet not found: 35a529702f924a5c9bf53c3c668c6094
I20251214 20:52:28.762025 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 385) took 1721 ms. Trace:
I20251214 20:52:28.762071 2564 raft_consensus.cc:493] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:28.762288 991 leader_election.cc:343] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.762476 2564 raft_consensus.cc:515] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.764130 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9da5e98c6f4843449886176e9fcfa8e1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:28.764400 1052 raft_consensus.cc:2468] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:28.765033 851 leader_election.cc:304] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.765395 2572 raft_consensus.cc:2804] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:28.765468 2572 raft_consensus.cc:493] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:28.765530 2572 raft_consensus.cc:3060] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:28.767298 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2807af0754914688bd7a090b1bc32eb1 consensus state has no leader
I20251214 20:52:28.767661 446 master_service.cc:954] ReplaceTablet: received request to replace tablet 25dcf491653147e7831217c1ec7a2e0c from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:28.769006 2572 raft_consensus.cc:515] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.770735 2572 leader_election.cc:290] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:28.771508 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9da5e98c6f4843449886176e9fcfa8e1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:28.771746 1052 raft_consensus.cc:3060] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.772081 1203 tablet_bootstrap.cc:492] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:28.772217 1203 ts_tablet_manager.cc:1403] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.004s
I20251214 20:52:28.773018 1203 raft_consensus.cc:359] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:28.773178 1203 raft_consensus.cc:385] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.773226 1203 raft_consensus.cc:740] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.773451 1203 consensus_queue.cc:260] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:28.760780 2643 tablet.cc:1722] T 8d72f55d2d2440d993aa205d96c356f1 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:28.774041 1203 ts_tablet_manager.cc:1434] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:28.774279 1203 tablet_bootstrap.cc:492] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:28.775063 1052 raft_consensus.cc:2468] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:28.775838 1203 tablet_bootstrap.cc:654] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.776118 851 leader_election.cc:304] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:28.776574 2572 raft_consensus.cc:2804] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:28.777145 2572 raft_consensus.cc:697] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:28.777489 2572 consensus_queue.cc:237] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.750201 712 leader_election.cc:304] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:28.752049 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "453cf734f8b34b59aa246d618ffeade5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:28.754938 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 98105bceda9b44bf893f82a4234a3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.770537 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9da5e98c6f4843449886176e9fcfa8e1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:28.742946 1982 ts_tablet_manager.cc:1929] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:28.781581 1982 log.cc:1199] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/1e31615d1277412eb8bd80451f2c4366
I20251214 20:52:28.781958 1982 ts_tablet_manager.cc:1950] T 1e31615d1277412eb8bd80451f2c4366 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
W20251214 20:52:28.781991 848 leader_election.cc:343] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.782696 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 378) took 1346 ms. Trace:
I20251214 20:52:28.782842 1982 rpcz_store.cc:276] 1214 20:52:27.436284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.436715 (+ 431us) service_pool.cc:225] Handling call
1214 20:52:28.663667 (+1226952us) ts_tablet_manager.cc:1151] Deleting tablet 1e31615d1277412eb8bd80451f2c4366
1214 20:52:28.663676 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.740204 (+ 76528us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.742846 (+ 2642us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.782677 (+ 39831us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1226810}
I20251214 20:52:28.783303 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 379) took 1346 ms. Trace:
I20251214 20:52:28.783409 1982 rpcz_store.cc:276] 1214 20:52:27.436427 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.436879 (+ 452us) service_pool.cc:225] Handling call
1214 20:52:28.783183 (+1346304us) ts_tablet_manager.cc:1151] Deleting tablet cea9423c40044e4cabb8bbd755a46688
1214 20:52:28.783193 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.783288 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1346196}
I20251214 20:52:28.783742 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 380) took 1210 ms. Trace:
I20251214 20:52:28.783839 1982 rpcz_store.cc:276] 1214 20:52:27.572939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.572994 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:28.783655 (+1210661us) ts_tablet_manager.cc:1151] Deleting tablet 1c5ee4114cb34540af2c5b332ebe82ab
1214 20:52:28.783663 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.783728 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206625}
I20251214 20:52:28.784163 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 383) took 1174 ms. Trace:
I20251214 20:52:28.784263 1982 rpcz_store.cc:276] 1214 20:52:27.610097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.610512 (+ 415us) service_pool.cc:225] Handling call
1214 20:52:28.784077 (+1173565us) ts_tablet_manager.cc:1151] Deleting tablet 9e1835374cfa48939d72476870accc3b
1214 20:52:28.784085 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.784151 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1173371}
I20251214 20:52:28.784439 2470 raft_consensus.cc:2804] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:28.784585 2470 raft_consensus.cc:697] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:28.784636 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 384) took 1174 ms. Trace:
I20251214 20:52:28.784811 1982 rpcz_store.cc:276] 1214 20:52:27.610180 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.610741 (+ 561us) service_pool.cc:225] Handling call
1214 20:52:28.784465 (+1173724us) ts_tablet_manager.cc:1151] Deleting tablet ebb26a5ae7224e9ca8a513c7b5013618
1214 20:52:28.784473 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.784518 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1173622}
I20251214 20:52:28.785343 1982 tablet_replica.cc:333] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:28.785339 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]) successfully deleted
I20251214 20:52:28.785742 1982 raft_consensus.cc:2243] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:28.785743 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet cea9423c40044e4cabb8bbd755a46688 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cea9423c40044e4cabb8bbd755a46688 already in progress: creating tablet
I20251214 20:52:28.786152 1982 raft_consensus.cc:2272] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:28.786264 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 9e1835374cfa48939d72476870accc3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e1835374cfa48939d72476870accc3b already in progress: creating tablet
W20251214 20:52:28.786608 439 catalog_manager.cc:4952] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ebb26a5ae7224e9ca8a513c7b5013618 because the tablet was not found. No further retry: Not found: Tablet not found: ebb26a5ae7224e9ca8a513c7b5013618
W20251214 20:52:28.786998 989 leader_election.cc:343] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.787138 989 leader_election.cc:304] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:28.787547 2561 raft_consensus.cc:2749] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:28.787779 1982 ts_tablet_manager.cc:1916] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.788321 2564 leader_election.cc:290] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:28.788935 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_6588026e2d974db9b72d2b81ce8310cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:28.788727 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9da5e98c6f4843449886176e9fcfa8e1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:28.789911 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6588026e2d974db9b72d2b81ce8310cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:28.790181 848 leader_election.cc:343] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.784874 2470 consensus_queue.cc:237] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.807499 453 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet e90d11d69b8d4fd49150aa67a8190e5b (table default.Table_c5bdb81886bf4361af53fd1a3b3b0b16 [id=b3df2580f4a74189aa06f021d6b4ce71]). Expected version 1 got 0
I20251214 20:52:28.807732 910 raft_consensus.cc:2468] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:28.807821 453 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:28.809350 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 9b1a937150ce4ecfa210c8465d682496 with delete_type TABLET_DATA_DELETED (replaced by tablet f1eb4c3aa654480caaa69dbd5047365c) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.809423 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_4abdf07a08814d65acc20b6c32bfd415"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:28.809597 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f2314d33fa474ca36dafe0a0eae5ae with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82068013e8471c9309bcf8e8604ece) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.809762 877 tablet_service.cc:1552] Processing DeleteTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:28.809971 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4abdf07a08814d65acc20b6c32bfd415 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.811218 455 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:28.811890 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_b9ef85d4768d42d28d635a7c3eac5f4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.812322 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9ef85d4768d42d28d635a7c3eac5f4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.816175 1036 tablet_service.cc:1505] Processing CreateTablet for tablet c249c096a41f4b4093eaf9afc6361249 (DEFAULT_TABLE table=default.Table_35def73a244b4885ad22656a1df2b697 [id=6694e71936c34466876fb63e0f337389]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.816572 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c249c096a41f4b4093eaf9afc6361249. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.817219 1202 tablet_bootstrap.cc:492] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:28.813714 2551 raft_consensus.cc:493] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.817538 1202 ts_tablet_manager.cc:1403] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.201s user 0.006s sys 0.001s
I20251214 20:52:28.817535 2551 raft_consensus.cc:515] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:28.817947 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 098dda4fe9d94fd8be2380d3f840cc92 (DEFAULT_TABLE table=default.Table_ac56c984d6f5476cb187409e4e7575b6 [id=1350143e22c44ac59f5ac65a3c2f5763]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.818265 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 098dda4fe9d94fd8be2380d3f840cc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.818365 2551 leader_election.cc:290] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:28.818930 1202 raft_consensus.cc:359] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.819362 1022 tablet_service.cc:1505] Processing CreateTablet for tablet cab1bed6016243bda0d9bd7f512884c8 (DEFAULT_TABLE table=default.Table_c3bedc14bbf14c5988f9c60ceacf9d95 [id=58ace009e5fc4eab93d8babc6330f58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.819711 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab1bed6016243bda0d9bd7f512884c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.828087 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0f7bd831b4474a90288ac7fa600436 (DEFAULT_TABLE table=default.Table_d2428076aa4e41dbb5b7cd6937e54a20 [id=c97bcc9ac37c4413bae618ee1019e069]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.828447 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0f7bd831b4474a90288ac7fa600436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.827322 877 tablet_service.cc:1505] Processing CreateTablet for tablet c249c096a41f4b4093eaf9afc6361249 (DEFAULT_TABLE table=default.Table_35def73a244b4885ad22656a1df2b697 [id=6694e71936c34466876fb63e0f337389]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.832181 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c249c096a41f4b4093eaf9afc6361249. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.834084 1856 ts_tablet_manager.cc:1929] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:28.834189 1856 log.cc:1199] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/4aa1a44452074ec28a1bd9c48a5f47cb
I20251214 20:52:28.827729 890 tablet_service.cc:1505] Processing CreateTablet for tablet cab1bed6016243bda0d9bd7f512884c8 (DEFAULT_TABLE table=default.Table_c3bedc14bbf14c5988f9c60ceacf9d95 [id=58ace009e5fc4eab93d8babc6330f58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.835554 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab1bed6016243bda0d9bd7f512884c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.835544 747 tablet_service.cc:1505] Processing CreateTablet for tablet cab1bed6016243bda0d9bd7f512884c8 (DEFAULT_TABLE table=default.Table_c3bedc14bbf14c5988f9c60ceacf9d95 [id=58ace009e5fc4eab93d8babc6330f58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.835852 1856 ts_tablet_manager.cc:1950] T 4aa1a44452074ec28a1bd9c48a5f47cb P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:28.835875 743 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0f7bd831b4474a90288ac7fa600436 (DEFAULT_TABLE table=default.Table_d2428076aa4e41dbb5b7cd6937e54a20 [id=c97bcc9ac37c4413bae618ee1019e069]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.835963 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab1bed6016243bda0d9bd7f512884c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.836138 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0f7bd831b4474a90288ac7fa600436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.836189 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 436) took 1394 ms. Trace:
I20251214 20:52:28.836265 1856 rpcz_store.cc:276] 1214 20:52:27.441702 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.442179 (+ 477us) service_pool.cc:225] Handling call
1214 20:52:28.634018 (+1191839us) ts_tablet_manager.cc:1151] Deleting tablet 4aa1a44452074ec28a1bd9c48a5f47cb
1214 20:52:28.634031 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.810827 (+176796us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.833966 (+ 23139us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:28.836170 (+ 2204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2,"tablet-delete.queue_time_us":1178126}
I20251214 20:52:28.836700 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 439) took 1263 ms. Trace:
I20251214 20:52:28.836763 1856 rpcz_store.cc:276] 1214 20:52:27.573664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.573723 (+ 59us) service_pool.cc:225] Handling call
1214 20:52:28.836572 (+1262849us) ts_tablet_manager.cc:1151] Deleting tablet 1c5ee4114cb34540af2c5b332ebe82ab
1214 20:52:28.836581 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.836686 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1262639}
I20251214 20:52:28.835544 749 tablet_service.cc:1505] Processing CreateTablet for tablet 098dda4fe9d94fd8be2380d3f840cc92 (DEFAULT_TABLE table=default.Table_ac56c984d6f5476cb187409e4e7575b6 [id=1350143e22c44ac59f5ac65a3c2f5763]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.839273 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 098dda4fe9d94fd8be2380d3f840cc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.839769 746 tablet_service.cc:1505] Processing CreateTablet for tablet c249c096a41f4b4093eaf9afc6361249 (DEFAULT_TABLE table=default.Table_35def73a244b4885ad22656a1df2b697 [id=6694e71936c34466876fb63e0f337389]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.840138 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c249c096a41f4b4093eaf9afc6361249. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.842060 991 leader_election.cc:304] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:28.842635 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06381efe75284e649fd6d1a9903cfa9b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.828718 880 tablet_service.cc:1505] Processing CreateTablet for tablet 098dda4fe9d94fd8be2380d3f840cc92 (DEFAULT_TABLE table=default.Table_ac56c984d6f5476cb187409e4e7575b6 [id=1350143e22c44ac59f5ac65a3c2f5763]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:28.843152 991 leader_election.cc:343] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.843494 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 098dda4fe9d94fd8be2380d3f840cc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.843529 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 4aa1a44452074ec28a1bd9c48a5f47cb (table default.Table_dbcc1409fc7e48cb804842a37eebaad2 [id=81c0278b4add47d9947b461866fe9eae]) successfully deleted
I20251214 20:52:28.844878 2551 raft_consensus.cc:2804] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:28.845006 2551 raft_consensus.cc:697] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:28.845260 2551 consensus_queue.cc:237] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.847769 885 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0f7bd831b4474a90288ac7fa600436 (DEFAULT_TABLE table=default.Table_d2428076aa4e41dbb5b7cd6937e54a20 [id=c97bcc9ac37c4413bae618ee1019e069]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.848316 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0f7bd831b4474a90288ac7fa600436. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.837018 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 441) took 1225 ms. Trace:
I20251214 20:52:28.853559 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06381efe75284e649fd6d1a9903cfa9b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:28.854564 1856 rpcz_store.cc:276] 1214 20:52:27.611710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.611789 (+ 79us) service_pool.cc:225] Handling call
1214 20:52:28.836937 (+1225148us) ts_tablet_manager.cc:1151] Deleting tablet 6d535d8f71b44c81bbba6937e804d900
1214 20:52:28.836944 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.837006 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224957}
W20251214 20:52:28.854741 989 leader_election.cc:343] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.855435 989 leader_election.cc:304] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:28.856227 2364 raft_consensus.cc:2749] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:28.857115 452 catalog_manager.cc:5654] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
W20251214 20:52:28.858036 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 6d535d8f71b44c81bbba6937e804d900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d535d8f71b44c81bbba6937e804d900 already in progress: creating tablet
I20251214 20:52:28.858430 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 442) took 1246 ms. Trace:
I20251214 20:52:28.858654 452 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 9da5e98c6f4843449886176e9fcfa8e1 (table default.Table_55f303d33ec84a269232fc1ba8f2d182 [id=5ded9fe825ae4308b5bd4b21c16634c8]). Expected version 1 got 0
I20251214 20:52:28.858927 452 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:28.859946 1856 rpcz_store.cc:276] 1214 20:52:27.611858 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.612023 (+ 165us) service_pool.cc:225] Handling call
1214 20:52:28.858300 (+1246277us) ts_tablet_manager.cc:1151] Deleting tablet 9e1835374cfa48939d72476870accc3b
1214 20:52:28.858311 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.858415 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1246171}
W20251214 20:52:28.860874 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 9e1835374cfa48939d72476870accc3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e1835374cfa48939d72476870accc3b already in progress: creating tablet
I20251214 20:52:28.861405 879 tablet_service.cc:1505] Processing CreateTablet for tablet 52f5af48440b4d569939c163f365f6fe (DEFAULT_TABLE table=default.Table_52200ef9aa154f64ae25af61268f6d6b [id=35d3089ba6ef4f28ada7749dc8fb5ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.861824 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 52f5af48440b4d569939c163f365f6fe (DEFAULT_TABLE table=default.Table_52200ef9aa154f64ae25af61268f6d6b [id=35d3089ba6ef4f28ada7749dc8fb5ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.862300 749 tablet_service.cc:1505] Processing CreateTablet for tablet 52f5af48440b4d569939c163f365f6fe (DEFAULT_TABLE table=default.Table_52200ef9aa154f64ae25af61268f6d6b [id=35d3089ba6ef4f28ada7749dc8fb5ceb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.861902 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 445) took 1214 ms. Trace:
I20251214 20:52:28.862747 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52f5af48440b4d569939c163f365f6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.862250 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 18e95ff0e0da4c7680f594ba31cdc3a1 (DEFAULT_TABLE table=default.Table_3377d29944b3480a88fd7c603ccd0a02 [id=de970ec073534ca59adcdfd3470ee87c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.862898 1856 rpcz_store.cc:276] 1214 20:52:27.647300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.647582 (+ 282us) service_pool.cc:225] Handling call
1214 20:52:28.861785 (+1214203us) ts_tablet_manager.cc:1151] Deleting tablet ed9790fbebda4f848291930bf1dd8174
1214 20:52:28.861794 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.861886 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213569}
I20251214 20:52:28.862758 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52f5af48440b4d569939c163f365f6fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:28.863430 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ed9790fbebda4f848291930bf1dd8174 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed9790fbebda4f848291930bf1dd8174 already in progress: creating tablet
I20251214 20:52:28.863665 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e95ff0e0da4c7680f594ba31cdc3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.869741 2572 consensus_queue.cc:1048] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:28.871106 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 448) took 1189 ms. Trace:
I20251214 20:52:28.871954 775 raft_consensus.cc:3060] T ba7952290e2c4746802cffeb14971da0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.872210 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 1e185d107619414a8bcc15ad496547dc
I20251214 20:52:28.872967 1856 rpcz_store.cc:276] 1214 20:52:27.682053 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.682210 (+ 157us) service_pool.cc:225] Handling call
1214 20:52:28.870988 (+1188778us) ts_tablet_manager.cc:1151] Deleting tablet 52c05204502a45e18c2b862e21c6f361
1214 20:52:28.870997 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.871088 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131733}
I20251214 20:52:28.873819 1856 tablet_replica.cc:333] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:28.874006 1856 raft_consensus.cc:2243] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:28.874490 878 tablet_service.cc:1505] Processing CreateTablet for tablet 18e95ff0e0da4c7680f594ba31cdc3a1 (DEFAULT_TABLE table=default.Table_3377d29944b3480a88fd7c603ccd0a02 [id=de970ec073534ca59adcdfd3470ee87c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.875355 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52f5af48440b4d569939c163f365f6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.875511 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e185d107619414a8bcc15ad496547dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:37530
W20251214 20:52:28.875885 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 52c05204502a45e18c2b862e21c6f361 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52c05204502a45e18c2b862e21c6f361 already in progress: creating tablet
I20251214 20:52:28.876037 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e185d107619414a8bcc15ad496547dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.874316 747 tablet_service.cc:1505] Processing CreateTablet for tablet 526f28f9aaad445081d9d69cc20ed7b3 (DEFAULT_TABLE table=default.Table_b87980e2da6a4052b109938fcf0527d8 [id=88d5807cc76646e3a2962b5d36413894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.876823 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 526f28f9aaad445081d9d69cc20ed7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.878285 884 tablet_service.cc:1505] Processing CreateTablet for tablet 526f28f9aaad445081d9d69cc20ed7b3 (DEFAULT_TABLE table=default.Table_b87980e2da6a4052b109938fcf0527d8 [id=88d5807cc76646e3a2962b5d36413894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.878736 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 526f28f9aaad445081d9d69cc20ed7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.880568 883 tablet_service.cc:1505] Processing CreateTablet for tablet 348d8dcef97042279bc0d5ff31d3c137 (DEFAULT_TABLE table=default.Table_baed953978784a8eb83ffe465a406dfb [id=73aa4322355f451babb2c25a159cb296]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.880942 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348d8dcef97042279bc0d5ff31d3c137. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:28.881001 848 consensus_peers.cc:597] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:28.883504 743 tablet_service.cc:1505] Processing CreateTablet for tablet 348d8dcef97042279bc0d5ff31d3c137 (DEFAULT_TABLE table=default.Table_baed953978784a8eb83ffe465a406dfb [id=73aa4322355f451babb2c25a159cb296]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.883960 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348d8dcef97042279bc0d5ff31d3c137. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.884622 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e95ff0e0da4c7680f594ba31cdc3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.887161 748 tablet_service.cc:1505] Processing CreateTablet for tablet edcf8857116d4394b66f5fa7fce343a5 (DEFAULT_TABLE table=default.Table_fc1d7cb6f8544f6285ec8792deb1da74 [id=8ce51bbd64ff4e1789fade77a6b0c13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.887526 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edcf8857116d4394b66f5fa7fce343a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.892709 746 tablet_service.cc:1505] Processing CreateTablet for tablet 18e95ff0e0da4c7680f594ba31cdc3a1 (DEFAULT_TABLE table=default.Table_3377d29944b3480a88fd7c603ccd0a02 [id=de970ec073534ca59adcdfd3470ee87c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.893214 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e95ff0e0da4c7680f594ba31cdc3a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.899925 1202 raft_consensus.cc:385] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.900084 1202 raft_consensus.cc:740] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.900419 1202 consensus_queue.cc:260] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.901033 1202 ts_tablet_manager.cc:1434] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.083s user 0.001s sys 0.001s
I20251214 20:52:28.901329 1202 tablet_bootstrap.cc:492] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:28.876534 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e185d107619414a8bcc15ad496547dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:28.879055 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 526f28f9aaad445081d9d69cc20ed7b3 (DEFAULT_TABLE table=default.Table_b87980e2da6a4052b109938fcf0527d8 [id=88d5807cc76646e3a2962b5d36413894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.902496 1856 pending_rounds.cc:70] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:28.902552 1856 pending_rounds.cc:77] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493766298849280 op_type: NO_OP noop_request { }
I20251214 20:52:28.879976 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 348d8dcef97042279bc0d5ff31d3c137 (DEFAULT_TABLE table=default.Table_baed953978784a8eb83ffe465a406dfb [id=73aa4322355f451babb2c25a159cb296]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.902647 1856 raft_consensus.cc:2889] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:28.902722 1856 raft_consensus.cc:2272] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.881625 1030 tablet_service.cc:1505] Processing CreateTablet for tablet edcf8857116d4394b66f5fa7fce343a5 (DEFAULT_TABLE table=default.Table_fc1d7cb6f8544f6285ec8792deb1da74 [id=8ce51bbd64ff4e1789fade77a6b0c13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.899770 2551 raft_consensus.cc:493] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.902937 2551 raft_consensus.cc:515] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.903081 1202 tablet_bootstrap.cc:654] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:28.903306 885 tablet_service.cc:1505] Processing CreateTablet for tablet edcf8857116d4394b66f5fa7fce343a5 (DEFAULT_TABLE table=default.Table_fc1d7cb6f8544f6285ec8792deb1da74 [id=8ce51bbd64ff4e1789fade77a6b0c13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.903959 2551 leader_election.cc:290] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.904135 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96dc4bdabb234c3fa4c26bd1633e783d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:28.904636 989 leader_election.cc:343] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.904584 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edcf8857116d4394b66f5fa7fce343a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.905853 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96dc4bdabb234c3fa4c26bd1633e783d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:28.906112 910 raft_consensus.cc:2468] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:28.906296 1856 ts_tablet_manager.cc:1916] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.906637 991 leader_election.cc:304] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:28.907698 2551 raft_consensus.cc:2804] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:28.907809 2551 raft_consensus.cc:493] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:28.907997 2551 raft_consensus.cc:3060] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.911260 1203 tablet_bootstrap.cc:492] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:28.911625 1203 ts_tablet_manager.cc:1403] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.137s user 0.005s sys 0.001s
I20251214 20:52:28.912657 1203 raft_consensus.cc:359] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:28.912773 2551 raft_consensus.cc:515] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.913643 2551 leader_election.cc:290] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.913039 1203 raft_consensus.cc:385] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:28.914103 1203 raft_consensus.cc:740] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:28.914451 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96dc4bdabb234c3fa4c26bd1633e783d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:28.914628 910 raft_consensus.cc:3060] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.914502 1203 consensus_queue.cc:260] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:28.915272 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "96dc4bdabb234c3fa4c26bd1633e783d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:28.917066 910 raft_consensus.cc:2468] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
W20251214 20:52:28.917610 989 leader_election.cc:343] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.918368 991 leader_election.cc:304] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:28.918738 2364 raft_consensus.cc:2804] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:28.918848 2364 raft_consensus.cc:697] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:28.919050 2364 consensus_queue.cc:237] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.925263 1103 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:28.918959 1203 ts_tablet_manager.cc:1434] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20251214 20:52:28.926363 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 176164us
I20251214 20:52:28.935416 1203 tablet_bootstrap.cc:492] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:28.936936 2635 tablet.cc:1722] T 144dee1a6d7344f2b79da6e79e67599e P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251214 20:52:28.940814 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 3915abc53b084854912a8e4119eefc7e
I20251214 20:52:28.942739 2551 raft_consensus.cc:493] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.942883 2551 raft_consensus.cc:515] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:28.942903 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 3915abc53b084854912a8e4119eefc7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.943507 2551 leader_election.cc:290] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:28.762084 1977 rpcz_store.cc:276] 1214 20:52:27.040474 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.040538 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:28.761926 (+1721388us) ts_tablet_manager.cc:1151] Deleting tablet 492770a7b8b74ba49315f974cc54eedb
1214 20:52:28.761935 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.762012 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1720827}
I20251214 20:52:28.944545 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dca369dc5f54a4ea1783e334c5670d6" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:28.944738 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 3915abc53b084854912a8e4119eefc7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:28.944401 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 492770a7b8b74ba49315f974cc54eedb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 492770a7b8b74ba49315f974cc54eedb already in progress: creating tablet
W20251214 20:52:28.945156 989 leader_election.cc:343] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.945791 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 386) took 1791 ms. Trace:
I20251214 20:52:28.945931 1977 rpcz_store.cc:276] 1214 20:52:27.154107 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.154207 (+ 100us) service_pool.cc:225] Handling call
1214 20:52:28.945661 (+1791454us) ts_tablet_manager.cc:1151] Deleting tablet 6a000951f61948ac85fdd431f3d162c2
1214 20:52:28.945673 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.945773 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1791194}
W20251214 20:52:28.946412 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 6a000951f61948ac85fdd431f3d162c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a000951f61948ac85fdd431f3d162c2 already in progress: creating tablet
I20251214 20:52:28.947539 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 3915abc53b084854912a8e4119eefc7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.949234 775 raft_consensus.cc:3060] T 00cf456eda464f5ba22d6893f73b5cba P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:28.950183 2634 tablet.cc:1722] T 144dee1a6d7344f2b79da6e79e67599e P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251214 20:52:28.953394 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 387) took 1693 ms. Trace:
I20251214 20:52:28.953250 2657 raft_consensus.cc:493] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:28.953505 1977 rpcz_store.cc:276] 1214 20:52:27.259853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.259982 (+ 129us) service_pool.cc:225] Handling call
1214 20:52:28.953266 (+1693284us) ts_tablet_manager.cc:1151] Deleting tablet 79e621f29c01448399d22615284fccd4
1214 20:52:28.953279 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.953378 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1685137}
W20251214 20:52:28.953984 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 79e621f29c01448399d22615284fccd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79e621f29c01448399d22615284fccd4 already in progress: creating tablet
I20251214 20:52:28.953862 2657 raft_consensus.cc:515] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:28.956327 2643 tablet.cc:1722] T 144dee1a6d7344f2b79da6e79e67599e P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251214 20:52:28.960259 1977 tablet_replica.cc:333] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:28.960474 1977 raft_consensus.cc:2243] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:28.960183 749 tablet_service.cc:1505] Processing CreateTablet for tablet 055b1cc69c9847ee81ee106fead48fa5 (DEFAULT_TABLE table=default.Table_bbd8d7f1a4b04eeeaf8ccd7e060b7476 [id=d6da680c34f440b2b0d1182a3f8ac19d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.960722 1977 raft_consensus.cc:2272] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:28.961583 1977 ts_tablet_manager.cc:1916] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:28.964445 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 055b1cc69c9847ee81ee106fead48fa5 (DEFAULT_TABLE table=default.Table_bbd8d7f1a4b04eeeaf8ccd7e060b7476 [id=d6da680c34f440b2b0d1182a3f8ac19d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.967532 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dca369dc5f54a4ea1783e334c5670d6" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:28.968539 991 leader_election.cc:343] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:28.968706 991 leader_election.cc:304] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:28.969130 2551 raft_consensus.cc:2749] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:28.970764 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d6384d1f74bc4ec9a1985c5d8655e4b6" } new_table_name: "default.Table_531eb871a88242c5b91423a2c6abc3b6" modify_external_catalogs: true
I20251214 20:52:28.972366 2657 leader_election.cc:290] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:28.974014 712 leader_election.cc:343] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: fe0c3c8a1d5b46cfacd35d3f0d82e432
W20251214 20:52:28.974999 713 leader_election.cc:343] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: fe0c3c8a1d5b46cfacd35d3f0d82e432
I20251214 20:52:28.975116 713 leader_election.cc:304] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:28.975672 2657 raft_consensus.cc:2749] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:28.981796 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 6b389a63a58d438db41a343db6e04a88
I20251214 20:52:28.982273 447 catalog_manager.cc:6724] ReplaceTablet: tablet 6b389a63a58d438db41a343db6e04a88 deleted and replaced by tablet f6d51d07f7e9450a844be9afddb60d03
I20251214 20:52:28.983438 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 055b1cc69c9847ee81ee106fead48fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.984602 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b389a63a58d438db41a343db6e04a88 with delete_type TABLET_DATA_DELETED (replaced by tablet f6d51d07f7e9450a844be9afddb60d03) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:28.984862 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b389a63a58d438db41a343db6e04a88 with delete_type TABLET_DATA_DELETED (replaced by tablet f6d51d07f7e9450a844be9afddb60d03) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:28.986191 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_b567effdb6834f7b87ce4aba9e555913"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:28.986884 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b567effdb6834f7b87ce4aba9e555913 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:28.989969 880 tablet_service.cc:1505] Processing CreateTablet for tablet 055b1cc69c9847ee81ee106fead48fa5 (DEFAULT_TABLE table=default.Table_bbd8d7f1a4b04eeeaf8ccd7e060b7476 [id=d6da680c34f440b2b0d1182a3f8ac19d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:28.990551 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 055b1cc69c9847ee81ee106fead48fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:28.994598 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b389a63a58d438db41a343db6e04a88 with delete_type TABLET_DATA_DELETED (replaced by tablet f6d51d07f7e9450a844be9afddb60d03) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.003823 446 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 25dcf491653147e7831217c1ec7a2e0c
I20251214 20:52:29.004383 446 catalog_manager.cc:6724] ReplaceTablet: tablet 25dcf491653147e7831217c1ec7a2e0c deleted and replaced by tablet fd3a49d6acae40d099cdd99981ce9d46
I20251214 20:52:29.006397 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 25dcf491653147e7831217c1ec7a2e0c with delete_type TABLET_DATA_DELETED (replaced by tablet fd3a49d6acae40d099cdd99981ce9d46) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:29.007221 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_680d6597d5f0419e83bfca87ec6d7787" } new_table_name: "default.Table_1be39fb576af4b2b9e2606b495619863" modify_external_catalogs: true
I20251214 20:52:29.010921 2645 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:29.011540 2645 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:29.006780 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 25dcf491653147e7831217c1ec7a2e0c with delete_type TABLET_DATA_DELETED (replaced by tablet fd3a49d6acae40d099cdd99981ce9d46) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.013710 1202 tablet_bootstrap.cc:492] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:29.013986 1202 ts_tablet_manager.cc:1403] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.003s
I20251214 20:52:29.014771 1202 raft_consensus.cc:359] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:29.016394 1202 raft_consensus.cc:385] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.016503 1202 raft_consensus.cc:740] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.016764 1202 consensus_queue.cc:260] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:29.017298 1202 ts_tablet_manager.cc:1434] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:29.016290 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 25dcf491653147e7831217c1ec7a2e0c with delete_type TABLET_DATA_DELETED (replaced by tablet fd3a49d6acae40d099cdd99981ce9d46) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.017617 1202 tablet_bootstrap.cc:492] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:29.022224 2439 sys_catalog.cc:565] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: configured and running, proceeding with master startup.
I20251214 20:52:29.025152 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet a88207c8cc5f4a6886de36c82c6b4c2f
I20251214 20:52:29.025367 1202 tablet_bootstrap.cc:654] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.026690 1977 ts_tablet_manager.cc:1929] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:29.027192 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet a88207c8cc5f4a6886de36c82c6b4c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.027758 1977 log.cc:1199] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/bf8643c7e4094f529cd55656c7b3edb6
I20251214 20:52:29.027887 880 tablet_service.cc:1552] Processing DeleteTablet for tablet a88207c8cc5f4a6886de36c82c6b4c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.028061 1977 ts_tablet_manager.cc:1950] T bf8643c7e4094f529cd55656c7b3edb6 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:29.028896 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 388) took 1638 ms. Trace:
I20251214 20:52:29.028982 1977 rpcz_store.cc:276] 1214 20:52:27.390531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.392375 (+ 1844us) service_pool.cc:225] Handling call
1214 20:52:28.960209 (+1567834us) ts_tablet_manager.cc:1151] Deleting tablet bf8643c7e4094f529cd55656c7b3edb6
1214 20:52:28.960219 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.990418 (+ 30199us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.026571 (+ 36153us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.028872 (+ 2301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1567577}
I20251214 20:52:29.029451 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet bf8643c7e4094f529cd55656c7b3edb6 (table default.Table_1bba0cb42a0c4aa986d55fdc492ffd46 [id=ce55ad7e80fe40e99895c52abb55e986]) successfully deleted
I20251214 20:52:29.032537 1977 tablet_replica.cc:333] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:29.032593 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_a133da1dda5b4d66b144f6b1fe90f605"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.033253 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a133da1dda5b4d66b144f6b1fe90f605 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.035168 748 tablet_service.cc:1505] Processing CreateTablet for tablet 06ebe9f717bf43108748605cf4f961b7 (DEFAULT_TABLE table=default.Table_ae43058f9a39467b9919a9e1c335602f [id=7c91475975864cac93bfeb4d13173f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.036207 743 tablet_service.cc:1552] Processing DeleteTablet for tablet a88207c8cc5f4a6886de36c82c6b4c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:29.041528 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 06ebe9f717bf43108748605cf4f961b7 (DEFAULT_TABLE table=default.Table_ae43058f9a39467b9919a9e1c335602f [id=7c91475975864cac93bfeb4d13173f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.043393 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet b2bea617954c475bac7e9b8719faa028
I20251214 20:52:29.034767 1982 ts_tablet_manager.cc:1929] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:29.046092 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet b2bea617954c475bac7e9b8719faa028 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.046763 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_80391b5a5aa9425794dfb7997c659009"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.047554 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80391b5a5aa9425794dfb7997c659009 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.048658 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06ebe9f717bf43108748605cf4f961b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.050634 1982 log.cc:1199] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/7dd657f3baff40bb8cb7f4648367f7a1
I20251214 20:52:29.048847 743 tablet_service.cc:1552] Processing DeleteTablet for tablet b2bea617954c475bac7e9b8719faa028 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:29.051015 1982 ts_tablet_manager.cc:1950] T 7dd657f3baff40bb8cb7f4648367f7a1 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:29.053140 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 385) took 1442 ms. Trace:
I20251214 20:52:29.053246 1982 rpcz_store.cc:276] 1214 20:52:27.610264 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.610861 (+ 597us) service_pool.cc:225] Handling call
1214 20:52:28.785303 (+1174442us) ts_tablet_manager.cc:1151] Deleting tablet 7dd657f3baff40bb8cb7f4648367f7a1
1214 20:52:28.785311 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:28.793492 (+ 8181us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.034612 (+241120us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.053120 (+ 18508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1174358}
I20251214 20:52:29.053679 1982 tablet_replica.cc:333] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:29.057847 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 7dd657f3baff40bb8cb7f4648367f7a1 (table default.Table_8b86c42f7826440982c88a8625559146 [id=c35009dabd4a42598506e52b3d3e493c]) successfully deleted
I20251214 20:52:29.060601 448 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet cbbfcef827ee4d548939aaf4783c8109
I20251214 20:52:29.062454 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet cbbfcef827ee4d548939aaf4783c8109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.067101 743 tablet_service.cc:1552] Processing DeleteTablet for tablet cbbfcef827ee4d548939aaf4783c8109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:29.053898 1982 raft_consensus.cc:2243] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:29.074883 1982 pending_rounds.cc:70] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:29.076581 1982 pending_rounds.cc:77] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493762520915968 op_type: NO_OP noop_request { }
I20251214 20:52:29.076859 1982 raft_consensus.cc:2889] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:29.077034 1982 raft_consensus.cc:2272] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.087877 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ded13abcaf9545afad1de571fd382831" } new_table_name: "default.Table_cdf0b37254924fa18a712b2bd472fb46" modify_external_catalogs: true
I20251214 20:52:29.088122 1206 tablet_bootstrap.cc:492] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:29.088338 1206 ts_tablet_manager.cc:1403] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.387s user 0.000s sys 0.013s
I20251214 20:52:29.088456 1982 ts_tablet_manager.cc:1916] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.089324 1206 raft_consensus.cc:359] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.093353 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_403e5819717949e8bffa1ed844a68964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.094136 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_403e5819717949e8bffa1ed844a68964 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.101120 1019 tablet_service.cc:1505] Processing CreateTablet for tablet a78664a3046d4c61aa20ba86ed9e486e (DEFAULT_TABLE table=default.Table_eaf150707b2442f1ae6f94ac564c6b87 [id=67577dd83c4045c992f1c574bd71edfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.101524 1206 raft_consensus.cc:385] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.104337 1206 raft_consensus.cc:740] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.100742 748 tablet_service.cc:1505] Processing CreateTablet for tablet a78664a3046d4c61aa20ba86ed9e486e (DEFAULT_TABLE table=default.Table_eaf150707b2442f1ae6f94ac564c6b87 [id=67577dd83c4045c992f1c574bd71edfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.105088 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78664a3046d4c61aa20ba86ed9e486e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:29.105655 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 016581519f4c4e0d832b49346d55b41c consensus state has no leader
I20251214 20:52:29.106124 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_7b56d13e748e405a821ea513405846fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.106884 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b56d13e748e405a821ea513405846fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.109357 1206 consensus_queue.cc:260] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.110184 449 catalog_manager.cc:5654] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:29.110654 449 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e90d11d69b8d4fd49150aa67a8190e5b (table default.Table_c5bdb81886bf4361af53fd1a3b3b0b16 [id=b3df2580f4a74189aa06f021d6b4ce71]). Expected version 1 got 0
I20251214 20:52:29.110885 449 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (table default.Table_fcafcf3c6ddb4457aea41320f7bf0b74 [id=4329d01e7f6147d0adad53fa86369b17]). Expected version 1 got 0
I20251214 20:52:29.111779 1206 ts_tablet_manager.cc:1434] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20251214 20:52:29.112082 1206 tablet_bootstrap.cc:492] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:29.112490 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_6fedc617de7e4cd985c616534bda854a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.113090 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fedc617de7e4cd985c616534bda854a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.114460 2652 raft_consensus.cc:493] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.114651 2652 raft_consensus.cc:515] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:29.116350 2652 leader_election.cc:290] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:29.116509 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bdcecba5534294aca1eb419f18e9d2" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:29.118892 1206 tablet_bootstrap.cc:654] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.033828 1977 raft_consensus.cc:2243] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:29.075647 884 tablet_service.cc:1505] Processing CreateTablet for tablet 06ebe9f717bf43108748605cf4f961b7 (DEFAULT_TABLE table=default.Table_ae43058f9a39467b9919a9e1c335602f [id=7c91475975864cac93bfeb4d13173f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.132687 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06ebe9f717bf43108748605cf4f961b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.133059 1977 raft_consensus.cc:2272] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.111584 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9f74fb2c6244a6887e3c922262bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:14 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.127672 749 tablet_service.cc:1505] Processing CreateTablet for tablet c6eab2148fbf412483f1a4723123f41f (DEFAULT_TABLE table=default.Table_7219830d8fb040debc2388f49091b802 [id=40d3f605109b43828c497df2bf4eef65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.115013 877 tablet_service.cc:1505] Processing CreateTablet for tablet a78664a3046d4c61aa20ba86ed9e486e (DEFAULT_TABLE table=default.Table_eaf150707b2442f1ae6f94ac564c6b87 [id=67577dd83c4045c992f1c574bd71edfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.046061 880 tablet_service.cc:1552] Processing DeleteTablet for tablet b2bea617954c475bac7e9b8719faa028 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.115813 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bdcecba5534294aca1eb419f18e9d2" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:29.131824 1202 tablet_bootstrap.cc:492] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:29.136000 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78664a3046d4c61aa20ba86ed9e486e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.136183 1202 ts_tablet_manager.cc:1403] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.003s
I20251214 20:52:29.136387 712 leader_election.cc:304] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:29.062621 878 tablet_service.cc:1552] Processing DeleteTablet for tablet cbbfcef827ee4d548939aaf4783c8109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.137223 2652 raft_consensus.cc:2749] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:29.141006 1021 tablet_service.cc:1505] Processing CreateTablet for tablet c6eab2148fbf412483f1a4723123f41f (DEFAULT_TABLE table=default.Table_7219830d8fb040debc2388f49091b802 [id=40d3f605109b43828c497df2bf4eef65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.142221 1977 ts_tablet_manager.cc:1916] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.147032 878 tablet_service.cc:1505] Processing CreateTablet for tablet c6eab2148fbf412483f1a4723123f41f (DEFAULT_TABLE table=default.Table_7219830d8fb040debc2388f49091b802 [id=40d3f605109b43828c497df2bf4eef65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.147969 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6eab2148fbf412483f1a4723123f41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.151938 743 tablet_service.cc:1505] Processing CreateTablet for tablet f8e98027ac3f4f36aded3aa5b47e6cad (DEFAULT_TABLE table=default.Table_d8643dc61ad94d65864a5eade80548fc [id=db0d9e33bf054daabddadd1248243609]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.156002 1202 raft_consensus.cc:359] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.160584 1202 raft_consensus.cc:385] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.162145 1202 raft_consensus.cc:740] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
W20251214 20:52:29.177088 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6302145a6e7b4297a58d9d7f3ad02dce consensus state has no leader
I20251214 20:52:29.177587 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_04ad0003a0524c668f9118432b1b816f" } new_table_name: "default.Table_398f36b3acb440d88213e38f63740ca5" modify_external_catalogs: true
I20251214 20:52:29.174252 1202 consensus_queue.cc:260] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.179384 451 master_service.cc:954] ReplaceTablet: received request to replace tablet b1f4ea72c6274569a98cab6afdc55d55 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:29.179957 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d8643dc61ad94d65864a5eade80548fc" } new_table_name: "default.Table_5c7e3a65102c45a1a23d29b95427fae5" modify_external_catalogs: true
I20251214 20:52:29.181310 775 raft_consensus.cc:3060] T 42bdcecba5534294aca1eb419f18e9d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:29.185684 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6eab2148fbf412483f1a4723123f41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.186327 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8e98027ac3f4f36aded3aa5b47e6cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.187466 1202 ts_tablet_manager.cc:1434] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.051s user 0.005s sys 0.000s
I20251214 20:52:29.192911 1202 tablet_bootstrap.cc:492] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:29.201041 2571 raft_consensus.cc:493] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.201261 2571 raft_consensus.cc:515] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.201972 2571 leader_election.cc:290] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:29.202797 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d9f74fb2c6244a6887e3c922262bbab" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:29.203490 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d9f74fb2c6244a6887e3c922262bbab" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:29.203547 848 leader_election.cc:343] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:29.203698 1050 raft_consensus.cc:2468] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:29.204653 989 consensus_peers.cc:597] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:29.204654 851 leader_election.cc:304] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:29.205099 2571 raft_consensus.cc:2804] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:29.205184 2571 raft_consensus.cc:493] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:29.205245 2571 raft_consensus.cc:3060] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:29.208267 2571 raft_consensus.cc:515] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.209358 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d9f74fb2c6244a6887e3c922262bbab" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:29.209527 1050 raft_consensus.cc:3060] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:29.212740 1050 raft_consensus.cc:2468] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:29.213263 851 leader_election.cc:304] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:29.213529 2656 raft_consensus.cc:2804] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:29.213637 2656 raft_consensus.cc:697] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:29.213871 2656 consensus_queue.cc:237] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.216553 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d9f74fb2c6244a6887e3c922262bbab" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:29.216517 1033 tablet_service.cc:1505] Processing CreateTablet for tablet f8e98027ac3f4f36aded3aa5b47e6cad (DEFAULT_TABLE table=default.Table_d8643dc61ad94d65864a5eade80548fc [id=db0d9e33bf054daabddadd1248243609]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:29.217139 848 leader_election.cc:343] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:29.217607 2571 leader_election.cc:290] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:29.221805 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_8cd3b56154ec4d9fac4ed571a2287cfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.222519 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cd3b56154ec4d9fac4ed571a2287cfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.224934 1202 tablet_bootstrap.cc:654] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.233237 2652 raft_consensus.cc:493] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.233388 2652 raft_consensus.cc:515] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.234082 2652 leader_election.cc:290] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:29.235265 712 leader_election.cc:343] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 98105bceda9b44bf893f82a4234a3c81
I20251214 20:52:29.236248 877 tablet_service.cc:1505] Processing CreateTablet for tablet f8e98027ac3f4f36aded3aa5b47e6cad (DEFAULT_TABLE table=default.Table_d8643dc61ad94d65864a5eade80548fc [id=db0d9e33bf054daabddadd1248243609]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:29.236725 713 leader_election.cc:343] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 98105bceda9b44bf893f82a4234a3c81
I20251214 20:52:29.236840 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8e98027ac3f4f36aded3aa5b47e6cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.236860 713 leader_election.cc:304] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:29.237085 2652 raft_consensus.cc:2749] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:29.237427 989 consensus_peers.cc:597] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:29.238981 1206 tablet_bootstrap.cc:492] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:29.239176 1206 ts_tablet_manager.cc:1403] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.127s user 0.001s sys 0.005s
I20251214 20:52:29.239166 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 831d3bdd6d6845b9aa18d4cff52c2f0b (DEFAULT_TABLE table=default.Table_59b03580c6ff4f28baeb68e68994d7dc [id=0c71124d6937494cbded6debf9e9e4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.239871 749 tablet_service.cc:1505] Processing CreateTablet for tablet 831d3bdd6d6845b9aa18d4cff52c2f0b (DEFAULT_TABLE table=default.Table_59b03580c6ff4f28baeb68e68994d7dc [id=0c71124d6937494cbded6debf9e9e4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.240314 1206 raft_consensus.cc:359] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.247932 1206 raft_consensus.cc:385] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.248123 1206 raft_consensus.cc:740] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.248447 1206 consensus_queue.cc:260] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.249264 1206 ts_tablet_manager.cc:1434] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251214 20:52:29.253682 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 831d3bdd6d6845b9aa18d4cff52c2f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.259377 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 526f28f9aaad445081d9d69cc20ed7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.261113 1203 tablet_bootstrap.cc:654] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.263072 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edcf8857116d4394b66f5fa7fce343a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.265172 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06ebe9f717bf43108748605cf4f961b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.267252 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348d8dcef97042279bc0d5ff31d3c137. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.271247 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 055b1cc69c9847ee81ee106fead48fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.272631 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78664a3046d4c61aa20ba86ed9e486e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.273819 1856 ts_tablet_manager.cc:1929] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:29.273955 1856 log.cc:1199] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/57f2314d33fa474ca36dafe0a0eae5ae
I20251214 20:52:29.274334 1856 ts_tablet_manager.cc:1950] T 57f2314d33fa474ca36dafe0a0eae5ae P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:29.275346 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6eab2148fbf412483f1a4723123f41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.275894 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8e98027ac3f4f36aded3aa5b47e6cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.279845 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 443) took 1643 ms. Trace:
I20251214 20:52:29.279965 1856 rpcz_store.cc:276] 1214 20:52:27.636691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.672849 (+ 36158us) service_pool.cc:225] Handling call
1214 20:52:28.873769 (+1200920us) ts_tablet_manager.cc:1151] Deleting tablet 57f2314d33fa474ca36dafe0a0eae5ae
1214 20:52:28.873779 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.270556 (+396777us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.273711 (+ 3155us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.279819 (+ 6108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1111790}
I20251214 20:52:29.280547 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 57f2314d33fa474ca36dafe0a0eae5ae (table default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c]) successfully deleted
I20251214 20:52:29.281531 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 453) took 1486 ms. Trace:
I20251214 20:52:29.282300 1856 rpcz_store.cc:276] 1214 20:52:27.795240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.795326 (+ 86us) service_pool.cc:225] Handling call
1214 20:52:29.281386 (+1486060us) ts_tablet_manager.cc:1151] Deleting tablet 2f9def8e4fd449599820a63845397750
1214 20:52:29.281397 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.281515 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1485851}
I20251214 20:52:29.283416 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 831d3bdd6d6845b9aa18d4cff52c2f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.283365 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 454) took 1470 ms. Trace:
I20251214 20:52:29.284112 1856 rpcz_store.cc:276] 1214 20:52:27.812760 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.812841 (+ 81us) service_pool.cc:225] Handling call
1214 20:52:29.283257 (+1470416us) ts_tablet_manager.cc:1151] Deleting tablet d171301941fa43908e05984c3b016007
1214 20:52:29.283267 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.283350 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1469195}
I20251214 20:52:29.284956 1856 tablet_replica.cc:333] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:29.285544 1856 raft_consensus.cc:2243] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:29.286337 1856 raft_consensus.cc:2272] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.287541 1856 ts_tablet_manager.cc:1916] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:29.287639 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2f9def8e4fd449599820a63845397750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f9def8e4fd449599820a63845397750 already in progress: creating tablet
W20251214 20:52:29.287994 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet d171301941fa43908e05984c3b016007 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d171301941fa43908e05984c3b016007 already in progress: creating tablet
I20251214 20:52:29.288249 1203 tablet_bootstrap.cc:492] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:29.294001 1203 ts_tablet_manager.cc:1403] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.359s user 0.011s sys 0.022s
I20251214 20:52:29.294931 1203 raft_consensus.cc:359] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.295445 1203 raft_consensus.cc:385] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.295553 1203 raft_consensus.cc:740] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.295837 1203 consensus_queue.cc:260] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.296638 1203 ts_tablet_manager.cc:1434] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:29.298346 1203 tablet_bootstrap.cc:492] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:29.299736 1203 tablet_bootstrap.cc:654] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.306743 1856 ts_tablet_manager.cc:1929] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:29.253688 1206 tablet_bootstrap.cc:492] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:29.309891 1977 ts_tablet_manager.cc:1929] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:29.310034 1977 log.cc:1199] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/ec85857f047e44d78835305ddcb70768
I20251214 20:52:29.311503 743 tablet_service.cc:1505] Processing CreateTablet for tablet ba322cb6ee084bc39788ddd439f5f15d (DEFAULT_TABLE table=default.Table_9196cd658f9041269835197751563391 [id=2f57c28c9c624a6ca1986c8831b85169]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.312778 878 tablet_service.cc:1505] Processing CreateTablet for tablet 831d3bdd6d6845b9aa18d4cff52c2f0b (DEFAULT_TABLE table=default.Table_59b03580c6ff4f28baeb68e68994d7dc [id=0c71124d6937494cbded6debf9e9e4be]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.313249 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 831d3bdd6d6845b9aa18d4cff52c2f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.315241 1977 ts_tablet_manager.cc:1950] T ec85857f047e44d78835305ddcb70768 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:29.317595 1856 log.cc:1199] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/17550eb0ee6b476f8a8c304451f678b2
I20251214 20:52:29.318148 1856 ts_tablet_manager.cc:1950] T 17550eb0ee6b476f8a8c304451f678b2 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:29.313614 1203 tablet_bootstrap.cc:492] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:29.318410 1203 ts_tablet_manager.cc:1403] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
W20251214 20:52:29.318857 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6302145a6e7b4297a58d9d7f3ad02dce (table default.Table_1be39fb576af4b2b9e2606b495619863 [id=54e0182cd3b9461aa593a11aa7709002], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6302145a6e7b4297a58d9d7f3ad02dce consensus state has no leader
I20251214 20:52:29.319254 1203 raft_consensus.cc:359] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.319543 1203 raft_consensus.cc:385] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.319636 1203 raft_consensus.cc:740] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.319940 1203 consensus_queue.cc:260] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.321691 2551 consensus_queue.cc:1048] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:29.322674 1203 ts_tablet_manager.cc:1434] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251214 20:52:29.324288 1203 tablet_bootstrap.cc:492] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:29.325192 1018 tablet_service.cc:1505] Processing CreateTablet for tablet ba322cb6ee084bc39788ddd439f5f15d (DEFAULT_TABLE table=default.Table_9196cd658f9041269835197751563391 [id=2f57c28c9c624a6ca1986c8831b85169]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.325686 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba322cb6ee084bc39788ddd439f5f15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.326807 1203 tablet_bootstrap.cc:654] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.332448 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 460) took 1438 ms. Trace:
I20251214 20:52:29.332583 1856 rpcz_store.cc:276] 1214 20:52:27.894333 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.894890 (+ 557us) service_pool.cc:225] Handling call
1214 20:52:29.284916 (+1390026us) ts_tablet_manager.cc:1151] Deleting tablet 17550eb0ee6b476f8a8c304451f678b2
1214 20:52:29.284924 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.303969 (+ 19045us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.306662 (+ 2693us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.332426 (+ 25764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1389811}
I20251214 20:52:29.333204 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 17550eb0ee6b476f8a8c304451f678b2 (table default.Table_6f66a268b27e4f9b8d09597224149553 [id=d36ff09def9741f8b440f7bdbfdc21a8]) successfully deleted
I20251214 20:52:29.333627 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 461) took 1439 ms. Trace:
I20251214 20:52:29.333735 1856 rpcz_store.cc:276] 1214 20:52:27.894459 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.895205 (+ 746us) service_pool.cc:225] Handling call
1214 20:52:29.333515 (+1438310us) ts_tablet_manager.cc:1151] Deleting tablet ca57963bcfb54eec9fcbed82757d8387
1214 20:52:29.333524 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.333598 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438147}
W20251214 20:52:29.334137 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet ca57963bcfb54eec9fcbed82757d8387 because the tablet was not found. No further retry: Not found: Tablet not found: ca57963bcfb54eec9fcbed82757d8387
I20251214 20:52:29.334187 748 tablet_service.cc:1505] Processing CreateTablet for tablet 76d16819e133434eb1c5a65fffb1f03f (DEFAULT_TABLE table=default.Table_0945dd902049447298aef35db32f33ba [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.334408 1856 tablet_replica.cc:333] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:29.334573 1856 raft_consensus.cc:2243] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:29.334836 1856 pending_rounds.cc:70] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:29.334884 1856 pending_rounds.cc:77] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493766349979648 op_type: NO_OP noop_request { }
I20251214 20:52:29.334983 1856 raft_consensus.cc:2889] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:29.335055 1856 raft_consensus.cc:2272] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.336133 1856 ts_tablet_manager.cc:1916] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.341719 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba322cb6ee084bc39788ddd439f5f15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.341995 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76d16819e133434eb1c5a65fffb1f03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.342568 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 76d16819e133434eb1c5a65fffb1f03f (DEFAULT_TABLE table=default.Table_0945dd902049447298aef35db32f33ba [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.343284 746 tablet_service.cc:1505] Processing CreateTablet for tablet 202b1d53cab644c98838b0ee81f29f22 (DEFAULT_TABLE table=default.Table_fefd2aebe38641a6892916b197d27abc [id=6094596f6c064c5bb1e20aa2b8767b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.343760 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 202b1d53cab644c98838b0ee81f29f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.346702 2652 raft_consensus.cc:493] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.346884 2652 raft_consensus.cc:515] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.347663 2652 leader_election.cc:290] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:29.349248 713 leader_election.cc:343] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 57f2314d33fa474ca36dafe0a0eae5ae
I20251214 20:52:29.350620 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76d16819e133434eb1c5a65fffb1f03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.351416 1982 ts_tablet_manager.cc:1929] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:29.351544 1982 log.cc:1199] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/b2e83bca8ed642b6a1b705ef70fde27f
I20251214 20:52:29.355490 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f2314d33fa474ca36dafe0a0eae5ae" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:29.355818 904 raft_consensus.cc:2393] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:29.356395 712 leader_election.cc:304] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:29.356920 2652 raft_consensus.cc:3060] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:29.360070 2652 raft_consensus.cc:2749] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:29.361970 1202 tablet_bootstrap.cc:492] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:29.362136 1202 ts_tablet_manager.cc:1403] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.171s user 0.011s sys 0.000s
I20251214 20:52:29.362319 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 202b1d53cab644c98838b0ee81f29f22 (DEFAULT_TABLE table=default.Table_fefd2aebe38641a6892916b197d27abc [id=6094596f6c064c5bb1e20aa2b8767b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.362717 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 202b1d53cab644c98838b0ee81f29f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.362994 748 tablet_service.cc:1505] Processing CreateTablet for tablet 6bd280c0d8f94893ba8eb1e97bb89cb0 (DEFAULT_TABLE table=default.Table_f6c14f9b22304e8aafbe1afab658ae71 [id=b6efd20d2ef54f799f88337bfc4f7980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.363583 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd280c0d8f94893ba8eb1e97bb89cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.369371 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 389) took 1978 ms. Trace:
I20251214 20:52:29.369500 1977 rpcz_store.cc:276] 1214 20:52:27.390672 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.392673 (+ 2001us) service_pool.cc:225] Handling call
1214 20:52:29.032481 (+1639808us) ts_tablet_manager.cc:1151] Deleting tablet ec85857f047e44d78835305ddcb70768
1214 20:52:29.032493 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.207832 (+175339us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.309751 (+101919us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.369337 (+ 59586us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":93,"tablet-delete.queue_time_us":1639677}
I20251214 20:52:29.369868 1977 tablet_replica.cc:333] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:29.370051 1977 raft_consensus.cc:2243] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:29.370208 1977 raft_consensus.cc:2272] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.370743 1977 ts_tablet_manager.cc:1916] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.371017 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 6bd280c0d8f94893ba8eb1e97bb89cb0 (DEFAULT_TABLE table=default.Table_f6c14f9b22304e8aafbe1afab658ae71 [id=b6efd20d2ef54f799f88337bfc4f7980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.371623 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd280c0d8f94893ba8eb1e97bb89cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.372350 1856 ts_tablet_manager.cc:1929] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:29.372447 749 tablet_service.cc:1505] Processing CreateTablet for tablet a12e8053c65343758d518ee840dbaee8 (DEFAULT_TABLE table=default.Table_02304abff69e46f68c7152bfc33cb44d [id=e815640c7ece4361800a22bc749915e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.372954 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12e8053c65343758d518ee840dbaee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.373425 1856 log.cc:1199] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/9b1a937150ce4ecfa210c8465d682496
I20251214 20:52:29.373946 1856 ts_tablet_manager.cc:1950] T 9b1a937150ce4ecfa210c8465d682496 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:29.376461 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 459) took 1482 ms. Trace:
I20251214 20:52:29.363667 880 tablet_service.cc:1505] Processing CreateTablet for tablet ba322cb6ee084bc39788ddd439f5f15d (DEFAULT_TABLE table=default.Table_9196cd658f9041269835197751563391 [id=2f57c28c9c624a6ca1986c8831b85169]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.377460 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba322cb6ee084bc39788ddd439f5f15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.379237 883 tablet_service.cc:1505] Processing CreateTablet for tablet 76d16819e133434eb1c5a65fffb1f03f (DEFAULT_TABLE table=default.Table_0945dd902049447298aef35db32f33ba [id=3b87708d1c7b460d925f7da030a1a23f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.381171 885 tablet_service.cc:1505] Processing CreateTablet for tablet 202b1d53cab644c98838b0ee81f29f22 (DEFAULT_TABLE table=default.Table_fefd2aebe38641a6892916b197d27abc [id=6094596f6c064c5bb1e20aa2b8767b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.379916 1033 tablet_service.cc:1505] Processing CreateTablet for tablet a12e8053c65343758d518ee840dbaee8 (DEFAULT_TABLE table=default.Table_02304abff69e46f68c7152bfc33cb44d [id=e815640c7ece4361800a22bc749915e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.380793 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 5b68367f902f4be386788a8d138b5762 (DEFAULT_TABLE table=default.Table_d5446dd7bdf14b76ab6136efd369dd12 [id=57e06d0c5ee94c0095db293af38b3eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.383457 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12e8053c65343758d518ee840dbaee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.383704 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b68367f902f4be386788a8d138b5762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.384320 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76d16819e133434eb1c5a65fffb1f03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.386883 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 202b1d53cab644c98838b0ee81f29f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.389094 1856 rpcz_store.cc:276] 1214 20:52:27.894169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.895184 (+ 1015us) service_pool.cc:225] Handling call
1214 20:52:29.334364 (+1439180us) ts_tablet_manager.cc:1151] Deleting tablet 9b1a937150ce4ecfa210c8465d682496
1214 20:52:29.334372 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.356423 (+ 22051us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.372265 (+ 15842us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.376441 (+ 4176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438688}
I20251214 20:52:29.390137 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 9b1a937150ce4ecfa210c8465d682496 (table default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]) successfully deleted
I20251214 20:52:29.392581 1202 raft_consensus.cc:359] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.394970 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet ec85857f047e44d78835305ddcb70768 (table default.Table_58a7d5bd289f408da592c143483b0329 [id=bb060058712440c7b1e5c40332a0263d]) successfully deleted
I20251214 20:52:29.395303 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 3f465cef46cc4e23982dced0d2ea9e36 (DEFAULT_TABLE table=default.Table_70cb422b9a3d439cba4cddcdd9b31c73 [id=74eb42fa8016492cad785dfa4d0088c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.400308 746 tablet_service.cc:1505] Processing CreateTablet for tablet 3f465cef46cc4e23982dced0d2ea9e36 (DEFAULT_TABLE table=default.Table_70cb422b9a3d439cba4cddcdd9b31c73 [id=74eb42fa8016492cad785dfa4d0088c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.400311 747 tablet_service.cc:1505] Processing CreateTablet for tablet 4b5360e7ece94fd2a5a6512db6c6c12f (DEFAULT_TABLE table=default.Table_2ca54bf356e24a1ebca6f4a155b83044 [id=7f407f5abea945498cf939a23b0f9319]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.401261 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b5360e7ece94fd2a5a6512db6c6c12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.402209 2604 consensus_queue.cc:1048] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:29.403380 989 consensus_peers.cc:597] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:29.405001 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c62ffb71fff43d4812eb86f40ce1c58 consensus state has no leader
I20251214 20:52:29.406081 879 tablet_service.cc:1505] Processing CreateTablet for tablet a12e8053c65343758d518ee840dbaee8 (DEFAULT_TABLE table=default.Table_02304abff69e46f68c7152bfc33cb44d [id=e815640c7ece4361800a22bc749915e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.406663 887 tablet_service.cc:1505] Processing CreateTablet for tablet 5b68367f902f4be386788a8d138b5762 (DEFAULT_TABLE table=default.Table_d5446dd7bdf14b76ab6136efd369dd12 [id=57e06d0c5ee94c0095db293af38b3eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.406934 748 tablet_service.cc:1505] Processing CreateTablet for tablet 5b68367f902f4be386788a8d138b5762 (DEFAULT_TABLE table=default.Table_d5446dd7bdf14b76ab6136efd369dd12 [id=57e06d0c5ee94c0095db293af38b3eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.407267 1202 raft_consensus.cc:385] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.407472 889 tablet_service.cc:1505] Processing CreateTablet for tablet 3f465cef46cc4e23982dced0d2ea9e36 (DEFAULT_TABLE table=default.Table_70cb422b9a3d439cba4cddcdd9b31c73 [id=74eb42fa8016492cad785dfa4d0088c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.409305 1202 raft_consensus.cc:740] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.409327 890 tablet_service.cc:1505] Processing CreateTablet for tablet 6bd280c0d8f94893ba8eb1e97bb89cb0 (DEFAULT_TABLE table=default.Table_f6c14f9b22304e8aafbe1afab658ae71 [id=b6efd20d2ef54f799f88337bfc4f7980]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.409874 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12e8053c65343758d518ee840dbaee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.409816 1202 consensus_queue.cc:260] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.410748 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b68367f902f4be386788a8d138b5762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.412568 962 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:29.412953 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd280c0d8f94893ba8eb1e97bb89cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.412968 1202 ts_tablet_manager.cc:1434] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.050s user 0.003s sys 0.000s
I20251214 20:52:29.413192 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f465cef46cc4e23982dced0d2ea9e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.412498 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b68367f902f4be386788a8d138b5762. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.424774 1982 ts_tablet_manager.cc:1950] T b2e83bca8ed642b6a1b705ef70fde27f P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:29.425119 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 4b5360e7ece94fd2a5a6512db6c6c12f (DEFAULT_TABLE table=default.Table_2ca54bf356e24a1ebca6f4a155b83044 [id=7f407f5abea945498cf939a23b0f9319]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.426651 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 8a818ca230564b93ac165d43ff22deb1 (DEFAULT_TABLE table=default.Table_79c0f2d73838484eba9470ccec3e90b9 [id=0a8ffa32e9ce4e3a8f9fa2afa4de69bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.428215 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 737b767718c54512a1547262db13b060 (DEFAULT_TABLE table=default.Table_1f214ce7548c43af80a74e233714ca0d [id=8f83a24aa5834c4b8c8bec3f0a8ac29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.430291 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 386) took 1819 ms. Trace:
I20251214 20:52:29.430521 1982 rpcz_store.cc:276] 1214 20:52:27.610353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.610959 (+ 606us) service_pool.cc:225] Handling call
1214 20:52:29.053612 (+1442653us) ts_tablet_manager.cc:1151] Deleting tablet b2e83bca8ed642b6a1b705ef70fde27f
1214 20:52:29.053623 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.209286 (+155663us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.351314 (+142028us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.430257 (+ 78943us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442564}
I20251214 20:52:29.431180 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 382) took 1821 ms. Trace:
I20251214 20:52:29.431917 1982 rpcz_store.cc:276] 1214 20:52:27.609987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.611952 (+ 1965us) service_pool.cc:225] Handling call
1214 20:52:29.430998 (+1819046us) ts_tablet_manager.cc:1151] Deleting tablet 6d535d8f71b44c81bbba6937e804d900
1214 20:52:29.431009 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.431161 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818865}
I20251214 20:52:29.432652 1982 tablet_replica.cc:333] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:29.433393 1982 raft_consensus.cc:2243] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:29.434546 1982 pending_rounds.cc:70] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:29.437156 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_47682ee6f46147558b5033379c2fe733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:29.437279 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 96157aae81624a4ba9fcf8c3f142911f (DEFAULT_TABLE table=default.Table_adb418bf04534b868c044988fb97e906 [id=d7e403a944e94d0d8c9d6ba115723ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.437546 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 95c84b7c32f34490a2869d036345e4c4 (DEFAULT_TABLE table=default.Table_e4ed29947e6544b9a23370d584496d4a [id=daf3f737290e4b789b9f81ed0f0fa375]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.437546 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 4a6c43ece40349fcb6c875c8b6d49409 (DEFAULT_TABLE table=default.Table_f86bf75508ca43ada4372af83098ae35 [id=ecd6b2f07b474967adf0662a8b89979b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:29.437884 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47682ee6f46147558b5033379c2fe733 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.438112 1025 tablet_service.cc:1505] Processing CreateTablet for tablet a5d28f09ad174936b47debace0599600 (DEFAULT_TABLE table=default.Table_93730665b1ab4686a0d3e54740fbddc8 [id=0b5581be22d345b3a0fd22c3dce2f207]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.438482 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 7e3e5f83fe6044219b6e34d7b290211f (DEFAULT_TABLE table=default.Table_acd41f528acd483cb2c37531c45283d6 [id=6cdbb231eb7e4d01b6023731b39f16a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.439050 1023 tablet_service.cc:1505] Processing CreateTablet for tablet bd17ca86d3a243d78d1f782c56f51a48 (DEFAULT_TABLE table=default.Table_70e73cbd7f9141f3ab482647b6118d52 [id=91a01b7a64464481b8ebe6331c0746da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.439352 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_cf7dd9af89cc4f06adec7de4838b9a32" } modify_external_catalogs: true
I20251214 20:52:29.439553 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_cf7dd9af89cc4f06adec7de4838b9a32" } modify_external_catalogs: true
I20251214 20:52:29.451684 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 465) took 1490 ms. Trace:
I20251214 20:52:29.455260 893 tablet_service.cc:1505] Processing CreateTablet for tablet bd17ca86d3a243d78d1f782c56f51a48 (DEFAULT_TABLE table=default.Table_70e73cbd7f9141f3ab482647b6118d52 [id=91a01b7a64464481b8ebe6331c0746da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.455821 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd17ca86d3a243d78d1f782c56f51a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.456030 882 tablet_service.cc:1505] Processing CreateTablet for tablet 737b767718c54512a1547262db13b060 (DEFAULT_TABLE table=default.Table_1f214ce7548c43af80a74e233714ca0d [id=8f83a24aa5834c4b8c8bec3f0a8ac29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.456496 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 737b767718c54512a1547262db13b060. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.457736 1202 tablet_bootstrap.cc:492] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:29.458268 881 tablet_service.cc:1505] Processing CreateTablet for tablet 7e3e5f83fe6044219b6e34d7b290211f (DEFAULT_TABLE table=default.Table_acd41f528acd483cb2c37531c45283d6 [id=6cdbb231eb7e4d01b6023731b39f16a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.460911 1856 rpcz_store.cc:276] 1214 20:52:27.960897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.960985 (+ 88us) service_pool.cc:225] Handling call
1214 20:52:29.389887 (+1428902us) ts_tablet_manager.cc:1151] Deleting tablet c3453fc996fb4d2b8cf0f80152098c43
1214 20:52:29.451234 (+ 61347us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.451637 (+ 403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1428612}
I20251214 20:52:29.455842 745 tablet_service.cc:1505] Processing CreateTablet for tablet 8a818ca230564b93ac165d43ff22deb1 (DEFAULT_TABLE table=default.Table_79c0f2d73838484eba9470ccec3e90b9 [id=0a8ffa32e9ce4e3a8f9fa2afa4de69bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.462723 750 tablet_service.cc:1505] Processing CreateTablet for tablet 4a6c43ece40349fcb6c875c8b6d49409 (DEFAULT_TABLE table=default.Table_f86bf75508ca43ada4372af83098ae35 [id=ecd6b2f07b474967adf0662a8b89979b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.463013 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a818ca230564b93ac165d43ff22deb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.463207 753 tablet_service.cc:1505] Processing CreateTablet for tablet 96157aae81624a4ba9fcf8c3f142911f (DEFAULT_TABLE table=default.Table_adb418bf04534b868c044988fb97e906 [id=d7e403a944e94d0d8c9d6ba115723ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.463615 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96157aae81624a4ba9fcf8c3f142911f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.461403 755 tablet_service.cc:1505] Processing CreateTablet for tablet a5d28f09ad174936b47debace0599600 (DEFAULT_TABLE table=default.Table_93730665b1ab4686a0d3e54740fbddc8 [id=0b5581be22d345b3a0fd22c3dce2f207]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.461478 742 tablet_service.cc:1505] Processing CreateTablet for tablet bd17ca86d3a243d78d1f782c56f51a48 (DEFAULT_TABLE table=default.Table_70e73cbd7f9141f3ab482647b6118d52 [id=91a01b7a64464481b8ebe6331c0746da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.461570 752 tablet_service.cc:1505] Processing CreateTablet for tablet dcecde5ed91c48689bed1d36bfd5725e (DEFAULT_TABLE table=default.Table_eba09dc20b824904b171bc39be41ea65 [id=a4ef8163c0a741f290495ef9015ae6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.461859 741 tablet_service.cc:1505] Processing CreateTablet for tablet 7e3e5f83fe6044219b6e34d7b290211f (DEFAULT_TABLE table=default.Table_acd41f528acd483cb2c37531c45283d6 [id=6cdbb231eb7e4d01b6023731b39f16a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.462436 744 tablet_service.cc:1505] Processing CreateTablet for tablet 95c84b7c32f34490a2869d036345e4c4 (DEFAULT_TABLE table=default.Table_e4ed29947e6544b9a23370d584496d4a [id=daf3f737290e4b789b9f81ed0f0fa375]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.467430 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5d28f09ad174936b47debace0599600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.468070 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c84b7c32f34490a2869d036345e4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.468863 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcecde5ed91c48689bed1d36bfd5725e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.471311 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd17ca86d3a243d78d1f782c56f51a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.475318 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6c43ece40349fcb6c875c8b6d49409. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.477733 1982 pending_rounds.cc:77] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493762711502848 op_type: NO_OP noop_request { }
I20251214 20:52:29.477873 1982 raft_consensus.cc:2889] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:29.477952 1982 raft_consensus.cc:2272] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.478579 1982 ts_tablet_manager.cc:1916] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.479266 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e3e5f83fe6044219b6e34d7b290211f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.479801 756 tablet_service.cc:1505] Processing CreateTablet for tablet 737b767718c54512a1547262db13b060 (DEFAULT_TABLE table=default.Table_1f214ce7548c43af80a74e233714ca0d [id=8f83a24aa5834c4b8c8bec3f0a8ac29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.481930 1203 tablet_bootstrap.cc:492] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:29.482103 1203 ts_tablet_manager.cc:1403] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.158s user 0.008s sys 0.018s
I20251214 20:52:29.482726 880 tablet_service.cc:1505] Processing CreateTablet for tablet 4a6c43ece40349fcb6c875c8b6d49409 (DEFAULT_TABLE table=default.Table_f86bf75508ca43ada4372af83098ae35 [id=ecd6b2f07b474967adf0662a8b89979b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.482956 1203 raft_consensus.cc:359] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.483157 1203 raft_consensus.cc:385] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.483207 1203 raft_consensus.cc:740] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.483441 1203 consensus_queue.cc:260] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.484179 1203 ts_tablet_manager.cc:1434] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:29.493861 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet b2e83bca8ed642b6a1b705ef70fde27f (table default.Table_04ad0003a0524c668f9118432b1b816f [id=954b3b4acdd94043b2608438d00483e7]) successfully deleted
W20251214 20:52:29.494174 439 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 44985.6us
W20251214 20:52:29.495244 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 6d535d8f71b44c81bbba6937e804d900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d535d8f71b44c81bbba6937e804d900 already in progress: creating tablet
I20251214 20:52:29.498155 452 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d7581ad6d39142f4a736c05d0e13811c" } modify_external_catalogs: true
I20251214 20:52:29.498381 452 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d7581ad6d39142f4a736c05d0e13811c" } modify_external_catalogs: true
I20251214 20:52:29.500371 885 tablet_service.cc:1505] Processing CreateTablet for tablet 8a818ca230564b93ac165d43ff22deb1 (DEFAULT_TABLE table=default.Table_79c0f2d73838484eba9470ccec3e90b9 [id=0a8ffa32e9ce4e3a8f9fa2afa4de69bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.501360 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6c43ece40349fcb6c875c8b6d49409. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.501648 447 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]). Expected version 1 got 0
I20251214 20:52:29.502034 447 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (table default.Table_fcafcf3c6ddb4457aea41320f7bf0b74 [id=4329d01e7f6147d0adad53fa86369b17]). Expected version 1 got 0
W20251214 20:52:29.496243 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c3453fc996fb4d2b8cf0f80152098c43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3453fc996fb4d2b8cf0f80152098c43 already in progress: creating tablet
I20251214 20:52:29.504418 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96157aae81624a4ba9fcf8c3f142911f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.505070 447 master_service.cc:954] ReplaceTablet: received request to replace tablet 69f6108792fa4c2a9744e3248c76251f from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:29.505606 447 master_service.cc:954] ReplaceTablet: received request to replace tablet 8e11723c042a4637839f1b13bed0935a from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:29.506574 448 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_7306395aef6240c39e095e0c8815130b [id=8cd62da3d57c4be4b5f10ace49d2e148] alter complete (version 5)
I20251214 20:52:29.507433 886 tablet_service.cc:1505] Processing CreateTablet for tablet 96157aae81624a4ba9fcf8c3f142911f (DEFAULT_TABLE table=default.Table_adb418bf04534b868c044988fb97e906 [id=d7e403a944e94d0d8c9d6ba115723ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.507687 889 tablet_service.cc:1505] Processing CreateTablet for tablet 4b5360e7ece94fd2a5a6512db6c6c12f (DEFAULT_TABLE table=default.Table_2ca54bf356e24a1ebca6f4a155b83044 [id=7f407f5abea945498cf939a23b0f9319]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.507858 891 tablet_service.cc:1505] Processing CreateTablet for tablet 95c84b7c32f34490a2869d036345e4c4 (DEFAULT_TABLE table=default.Table_e4ed29947e6544b9a23370d584496d4a [id=daf3f737290e4b789b9f81ed0f0fa375]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.508060 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c84b7c32f34490a2869d036345e4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.508948 448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d46e98eafc574e9a92bd66399d44b0fb" } modify_external_catalogs: true
I20251214 20:52:29.508989 1036 tablet_service.cc:1505] Processing CreateTablet for tablet dcecde5ed91c48689bed1d36bfd5725e (DEFAULT_TABLE table=default.Table_eba09dc20b824904b171bc39be41ea65 [id=a4ef8163c0a741f290495ef9015ae6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.509140 448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d46e98eafc574e9a92bd66399d44b0fb" } modify_external_catalogs: true
I20251214 20:52:29.510526 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f465cef46cc4e23982dced0d2ea9e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.512481 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 06381efe75284e649fd6d1a9903cfa9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.512560 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 7947b48d8e2e423da187cadf176f38ce (DEFAULT_TABLE table=default.Table_7694eee14c2346a9a71802bdf0897011 [id=9f14a0a266b947559b2173b20ead5ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.513167 876 tablet_service.cc:1505] Processing CreateTablet for tablet a5d28f09ad174936b47debace0599600 (DEFAULT_TABLE table=default.Table_93730665b1ab4686a0d3e54740fbddc8 [id=0b5581be22d345b3a0fd22c3dce2f207]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.514297 2551 consensus_queue.cc:1048] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:29.515053 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e3e5f83fe6044219b6e34d7b290211f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.515322 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5d28f09ad174936b47debace0599600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.516252 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7947b48d8e2e423da187cadf176f38ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.519233 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 737b767718c54512a1547262db13b060. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.520227 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_da414f75240a40e6b20b644312217eae" } modify_external_catalogs: true
I20251214 20:52:29.520481 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_da414f75240a40e6b20b644312217eae" } modify_external_catalogs: true
I20251214 20:52:29.521585 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd17ca86d3a243d78d1f782c56f51a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.523350 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a818ca230564b93ac165d43ff22deb1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:29.525404 446 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f11834cabdad445b8e53cd237d146282 consensus state has no leader
I20251214 20:52:29.527293 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b5360e7ece94fd2a5a6512db6c6c12f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:29.527556 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8e98027ac3f4f36aded3aa5b47e6cad consensus state has no leader
I20251214 20:52:29.530807 1856 tablet_replica.cc:333] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:29.531085 1856 raft_consensus.cc:2243] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:29.531414 1856 pending_rounds.cc:70] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:29.531456 1856 pending_rounds.cc:77] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493766224400384 op_type: NO_OP noop_request { }
I20251214 20:52:29.531476 450 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8163ed145f64447c8eb6fc0f8d0a1d4d" } modify_external_catalogs: true
I20251214 20:52:29.531560 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 9d56346fa26247f38ef716a9a8afac1a (DEFAULT_TABLE table=default.Table_debb0f541d1b4ca3bc00c5e2c81c8428 [id=3b70fc138995441896f4ec7341fb69d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.531714 450 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8163ed145f64447c8eb6fc0f8d0a1d4d" } modify_external_catalogs: true
I20251214 20:52:29.532003 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d56346fa26247f38ef716a9a8afac1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.533283 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_e2a035220b924122a538db32c77a683b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.533962 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2a035220b924122a538db32c77a683b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.531576 1856 raft_consensus.cc:2889] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:29.535367 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_1dc40537749846a8a77948884d646257"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:29.535509 1856 raft_consensus.cc:2272] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:29.535914 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dc40537749846a8a77948884d646257 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.536164 1856 ts_tablet_manager.cc:1916] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.538341 446 catalog_manager.cc:5654] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:29.541786 1203 tablet_bootstrap.cc:492] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:29.548206 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_5de6830aaec645f2992e4cfaceb20c27 [id=9553b46856c94003a293135c814a7df7]). Expected version 2 got 0
I20251214 20:52:29.551100 1203 tablet_bootstrap.cc:654] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.551829 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_19a53df5559b4eb5bfc7b41b54be081b" } new_table_name: "default.Table_c404f3a3b71f4899a233f864e78a7d13" modify_external_catalogs: true
I20251214 20:52:29.551800 446 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3a23561b9f4e48bbbd796e14fafec086 (table default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3 [id=f1338e81f4ec4dce87130f2ddbe98041]). Expected version 1 got 0
I20251214 20:52:29.554387 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e3e5f83fe6044219b6e34d7b290211f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.554911 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcecde5ed91c48689bed1d36bfd5725e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.561282 747 tablet_service.cc:1505] Processing CreateTablet for tablet 7947b48d8e2e423da187cadf176f38ce (DEFAULT_TABLE table=default.Table_7694eee14c2346a9a71802bdf0897011 [id=9f14a0a266b947559b2173b20ead5ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.561820 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7947b48d8e2e423da187cadf176f38ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.568014 745 tablet_service.cc:1505] Processing CreateTablet for tablet 9d56346fa26247f38ef716a9a8afac1a (DEFAULT_TABLE table=default.Table_debb0f541d1b4ca3bc00c5e2c81c8428 [id=3b70fc138995441896f4ec7341fb69d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.568526 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d56346fa26247f38ef716a9a8afac1a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:29.570917 2670 log.cc:927] Time spent T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20251214 20:52:29.572685 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 737b767718c54512a1547262db13b060. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.578372 892 tablet_service.cc:1505] Processing CreateTablet for tablet 9d56346fa26247f38ef716a9a8afac1a (DEFAULT_TABLE table=default.Table_debb0f541d1b4ca3bc00c5e2c81c8428 [id=3b70fc138995441896f4ec7341fb69d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.579555 2571 raft_consensus.cc:493] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.579711 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a818ca230564b93ac165d43ff22deb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.580190 2571 raft_consensus.cc:515] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.580971 2571 leader_election.cc:290] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:29.582685 1977 ts_tablet_manager.cc:1929] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:29.582841 1977 log.cc:1199] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/4aa1a44452074ec28a1bd9c48a5f47cb
I20251214 20:52:29.584493 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 1dca369dc5f54a4ea1783e334c5670d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.585422 1202 tablet_bootstrap.cc:654] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.585685 1206 tablet_bootstrap.cc:654] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.563087 1856 ts_tablet_manager.cc:1929] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:29.583222 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "876d2a1f1d1c4f6882a5857b43568ee5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:29.586750 1050 raft_consensus.cc:2468] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:29.587445 851 leader_election.cc:304] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:29.587592 879 tablet_service.cc:1505] Processing CreateTablet for tablet dcecde5ed91c48689bed1d36bfd5725e (DEFAULT_TABLE table=default.Table_eba09dc20b824904b171bc39be41ea65 [id=a4ef8163c0a741f290495ef9015ae6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.587981 1856 log.cc:1199] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/58ac06f852fc4abea5cb213926002ca1
I20251214 20:52:29.588418 1856 ts_tablet_manager.cc:1950] T 58ac06f852fc4abea5cb213926002ca1 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:29.588671 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d56346fa26247f38ef716a9a8afac1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.589740 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 466) took 1628 ms. Trace:
I20251214 20:52:29.589908 1856 rpcz_store.cc:276] 1214 20:52:27.961035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.961306 (+ 271us) service_pool.cc:225] Handling call
1214 20:52:29.461899 (+1500593us) ts_tablet_manager.cc:1151] Deleting tablet 58ac06f852fc4abea5cb213926002ca1
1214 20:52:29.530736 (+ 68837us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.559861 (+ 29125us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.562975 (+ 3114us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.589719 (+ 26744us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1500478}
I20251214 20:52:29.590607 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 58ac06f852fc4abea5cb213926002ca1 (table default.Table_3369f89218b140e7a52da8221726f4e1 [id=e901e2802c854b4f829e59e00aa90040]) successfully deleted
I20251214 20:52:29.596788 1203 tablet_bootstrap.cc:492] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:29.597400 1203 ts_tablet_manager.cc:1403] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.004s
I20251214 20:52:29.598754 1203 raft_consensus.cc:359] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.600317 1977 ts_tablet_manager.cc:1950] T 4aa1a44452074ec28a1bd9c48a5f47cb P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:29.603618 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "876d2a1f1d1c4f6882a5857b43568ee5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:29.604349 848 leader_election.cc:343] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:29.605283 2571 raft_consensus.cc:2804] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:29.605409 2571 raft_consensus.cc:493] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:29.605508 2571 raft_consensus.cc:3060] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:29.602252 883 tablet_service.cc:1505] Processing CreateTablet for tablet 7947b48d8e2e423da187cadf176f38ce (DEFAULT_TABLE table=default.Table_7694eee14c2346a9a71802bdf0897011 [id=9f14a0a266b947559b2173b20ead5ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.609691 1203 raft_consensus.cc:385] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.609927 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6c43ece40349fcb6c875c8b6d49409. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.610105 1203 raft_consensus.cc:740] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.602072 1856 tablet_replica.cc:333] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:29.610643 1856 raft_consensus.cc:2243] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:29.611289 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 390) took 2162 ms. Trace:
I20251214 20:52:29.611431 1977 rpcz_store.cc:276] 1214 20:52:27.438690 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.439183 (+ 493us) service_pool.cc:225] Handling call
1214 20:52:29.369803 (+1930620us) ts_tablet_manager.cc:1151] Deleting tablet 4aa1a44452074ec28a1bd9c48a5f47cb
1214 20:52:29.369816 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.446507 (+ 76691us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.582580 (+136073us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.601275 (+ 18695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1930462}
I20251214 20:52:29.612069 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 392) took 2173 ms. Trace:
I20251214 20:52:29.612174 1977 rpcz_store.cc:276] 1214 20:52:27.438897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.438951 (+ 54us) service_pool.cc:225] Handling call
1214 20:52:29.611850 (+2172899us) ts_tablet_manager.cc:1151] Deleting tablet cea9423c40044e4cabb8bbd755a46688
1214 20:52:29.611869 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.612020 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2172456}
I20251214 20:52:29.612493 1977 tablet_replica.cc:333] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:29.612726 1977 raft_consensus.cc:2243] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:29.612915 1977 raft_consensus.cc:2272] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.613095 1856 pending_rounds.cc:70] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: Trying to abort 1 pending ops.
I20251214 20:52:29.613456 1856 pending_rounds.cc:77] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493768073306112 op_type: NO_OP noop_request { }
I20251214 20:52:29.613531 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 4aa1a44452074ec28a1bd9c48a5f47cb (table default.Table_dbcc1409fc7e48cb804842a37eebaad2 [id=81c0278b4add47d9947b461866fe9eae]) successfully deleted
I20251214 20:52:29.613590 1977 ts_tablet_manager.cc:1916] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.613809 1856 raft_consensus.cc:2889] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20251214 20:52:29.613835 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet cea9423c40044e4cabb8bbd755a46688 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cea9423c40044e4cabb8bbd755a46688 already in progress: creating tablet
I20251214 20:52:29.613910 1856 raft_consensus.cc:2272] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.610997 1203 consensus_queue.cc:260] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.615290 2656 raft_consensus.cc:493] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.615435 2656 raft_consensus.cc:515] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.619087 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d949abef1cb41a9905482d81a98d2c1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:29.619318 1052 raft_consensus.cc:2468] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:29.619750 851 leader_election.cc:304] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:29.620004 2564 raft_consensus.cc:2804] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:29.620067 2564 raft_consensus.cc:493] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:29.620124 2564 raft_consensus.cc:3060] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:29.620461 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b5360e7ece94fd2a5a6512db6c6c12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.620952 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c84b7c32f34490a2869d036345e4c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.621062 1203 ts_tablet_manager.cc:1434] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20251214 20:52:29.623010 1856 ts_tablet_manager.cc:1916] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.627074 2564 raft_consensus.cc:515] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.627789 2564 leader_election.cc:290] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:29.628281 2656 leader_election.cc:290] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:29.628326 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d949abef1cb41a9905482d81a98d2c1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:29.629446 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d949abef1cb41a9905482d81a98d2c1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:29.629622 1052 raft_consensus.cc:3060] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:29.632236 1052 raft_consensus.cc:2468] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:29.628953 1203 tablet_bootstrap.cc:492] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:29.632530 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96157aae81624a4ba9fcf8c3f142911f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.632511 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d949abef1cb41a9905482d81a98d2c1" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:29.634910 848 leader_election.cc:343] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:29.635262 848 leader_election.cc:343] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:29.636452 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5d28f09ad174936b47debace0599600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.638970 1982 ts_tablet_manager.cc:1929] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:29.643658 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcecde5ed91c48689bed1d36bfd5725e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.646697 851 leader_election.cc:304] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:29.647671 2656 raft_consensus.cc:2804] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:29.647784 2656 raft_consensus.cc:697] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:29.648016 2656 consensus_queue.cc:237] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.649559 1203 tablet_bootstrap.cc:654] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.657734 1203 tablet_bootstrap.cc:492] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:29.658427 1203 ts_tablet_manager.cc:1403] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20251214 20:52:29.659634 1203 raft_consensus.cc:359] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.663810 1203 raft_consensus.cc:385] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.664078 1203 raft_consensus.cc:740] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.664644 1203 consensus_queue.cc:260] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.668316 1203 ts_tablet_manager.cc:1434] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251214 20:52:29.669150 1203 tablet_bootstrap.cc:492] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:29.669579 1982 log.cc:1199] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/242a77ae57174de48f4eaf43384ee0ba
I20251214 20:52:29.662709 1856 ts_tablet_manager.cc:1929] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:29.670994 1856 log.cc:1199] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/98b7cac3930a4a9eb16ac413c8f1e092
I20251214 20:52:29.671658 1856 ts_tablet_manager.cc:1950] T 98b7cac3930a4a9eb16ac413c8f1e092 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:29.670046 1982 ts_tablet_manager.cc:1950] T 242a77ae57174de48f4eaf43384ee0ba P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:29.675053 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 387) took 2064 ms. Trace:
I20251214 20:52:29.675220 1982 rpcz_store.cc:276] 1214 20:52:27.610455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.613411 (+ 2956us) service_pool.cc:225] Handling call
1214 20:52:29.432598 (+1819187us) ts_tablet_manager.cc:1151] Deleting tablet 242a77ae57174de48f4eaf43384ee0ba
1214 20:52:29.432607 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.625581 (+192974us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.638869 (+ 13288us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.675035 (+ 36166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818965}
I20251214 20:52:29.675742 1203 tablet_bootstrap.cc:654] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.675737 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 242a77ae57174de48f4eaf43384ee0ba (table default.Table_9ad469dea994477983bfe62eef37e334 [id=b1be575d16354425ab4f7e8e62847820]) successfully deleted
I20251214 20:52:29.675369 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 467) took 1714 ms. Trace:
I20251214 20:52:29.677034 1856 rpcz_store.cc:276] 1214 20:52:27.961149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.963229 (+ 2080us) service_pool.cc:225] Handling call
1214 20:52:29.601826 (+1638597us) ts_tablet_manager.cc:1151] Deleting tablet 98b7cac3930a4a9eb16ac413c8f1e092
1214 20:52:29.601928 (+ 102us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.650753 (+ 48825us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.662566 (+ 11813us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.675344 (+ 12778us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638344}
I20251214 20:52:29.677783 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 472) took 1511 ms. Trace:
I20251214 20:52:29.694530 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7947b48d8e2e423da187cadf176f38ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.694664 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 18e119bc44874aecb4e83c8d6040c8cd
I20251214 20:52:29.696300 452 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet d2a57409d4f54f17be792e6f3f17d7b7
I20251214 20:52:29.698401 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 8e11723c042a4637839f1b13bed0935a
I20251214 20:52:29.698891 447 catalog_manager.cc:6724] ReplaceTablet: tablet 8e11723c042a4637839f1b13bed0935a deleted and replaced by tablet 6e005feeb95b48b9ad33612ab81d5bce
I20251214 20:52:29.699440 448 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 0b3e0a3bab7a47cb8b26f2094f5a1680
I20251214 20:52:29.700364 1856 rpcz_store.cc:276] 1214 20:52:28.166684 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.166842 (+ 158us) service_pool.cc:225] Handling call
1214 20:52:29.677627 (+1510785us) ts_tablet_manager.cc:1151] Deleting tablet a0976ffcc649461fac74e20b5e402275
1214 20:52:29.677638 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.677765 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1510524}
I20251214 20:52:29.700866 1856 tablet_replica.cc:333] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:29.701093 1856 raft_consensus.cc:2243] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:29.702193 2705 raft_consensus.cc:493] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.702334 2705 raft_consensus.cc:515] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.704519 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet f98be57ce1b14f20a3afcb79979b6c22
I20251214 20:52:29.706470 1856 raft_consensus.cc:2272] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.707329 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet f7f06eded6c340c58101fcc644ebcf50
I20251214 20:52:29.707295 749 tablet_service.cc:1552] Processing DeleteTablet for tablet f98be57ce1b14f20a3afcb79979b6c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:29.707952 1856 ts_tablet_manager.cc:1916] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.708117 450 master_service.cc:954] ReplaceTablet: received request to replace tablet 292a28b12df246a8bc855d81c530746e from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:29.711282 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e11723c042a4637839f1b13bed0935a with delete_type TABLET_DATA_DELETED (replaced by tablet 6e005feeb95b48b9ad33612ab81d5bce) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:29.711517 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:29.711545 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f891db00595b49ca9e6ae64721e4d744 consensus state has no leader
I20251214 20:52:29.711697 752 tablet_service.cc:1552] Processing DeleteTablet for tablet f7f06eded6c340c58101fcc644ebcf50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:29.713147 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_4f6c749899d64d8884fd2ac83cfb13fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:29.703085 2705 leader_election.cc:290] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:29.713944 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f6c749899d64d8884fd2ac83cfb13fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.710777 1203 tablet_bootstrap.cc:492] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:29.714913 1203 ts_tablet_manager.cc:1403] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20251214 20:52:29.715981 1203 raft_consensus.cc:359] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.716174 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_040e46c43e8446f48a60e34f0892bf57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:29.716513 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d2428076aa4e41dbb5b7cd6937e54a20" } new_table_name: "default.Table_d8b86c6bc911454fac28b6384cf9ff3c" modify_external_catalogs: true
W20251214 20:52:29.716857 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_040e46c43e8446f48a60e34f0892bf57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.717203 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 390) took 2065 ms. Trace:
I20251214 20:52:29.717371 1982 rpcz_store.cc:276] 1214 20:52:27.651669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.664374 (+ 12705us) service_pool.cc:225] Handling call
1214 20:52:29.717054 (+2052680us) ts_tablet_manager.cc:1151] Deleting tablet ed9790fbebda4f848291930bf1dd8174
1214 20:52:29.717066 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.717183 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2052313}
I20251214 20:52:29.717978 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 393) took 2035 ms. Trace:
I20251214 20:52:29.718392 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_b3dbc2e42cc64d9d96c48ecc1b474480"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.718969 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3dbc2e42cc64d9d96c48ecc1b474480 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.719262 1982 rpcz_store.cc:276] 1214 20:52:27.682337 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.682390 (+ 53us) service_pool.cc:225] Handling call
1214 20:52:29.717874 (+2035484us) ts_tablet_manager.cc:1151] Deleting tablet 52c05204502a45e18c2b862e21c6f361
1214 20:52:29.717883 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.717962 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035297}
I20251214 20:52:29.720060 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "492770a7b8b74ba49315f974cc54eedb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:29.720288 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_eec6b34ea20f4ef096ef8df379a3bddb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.720796 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eec6b34ea20f4ef096ef8df379a3bddb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:29.720753 989 leader_election.cc:343] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:29.722203 1203 raft_consensus.cc:385] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.722357 1203 raft_consensus.cc:740] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.722613 1203 consensus_queue.cc:260] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.723024 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_f8b17194060c4627a7770e45799e082b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:29.723557 1203 ts_tablet_manager.cc:1434] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20251214 20:52:29.723558 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8b17194060c4627a7770e45799e082b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.723526 447 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ee7f67ff1a1c4014a1cc4f9cf7f6613b" } new_table_name: "default.Table_9098a11909c844dca6d4e80a284aa2e8" modify_external_catalogs: true
I20251214 20:52:29.723979 1203 tablet_bootstrap.cc:492] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:29.724797 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet ed9790fbebda4f848291930bf1dd8174 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed9790fbebda4f848291930bf1dd8174 already in progress: creating tablet
I20251214 20:52:29.725081 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:34819: OK, instance_id { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" instance_seqno: 1765745545713890 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251214 20:52:29.725310 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251214 20:52:29.727671 453 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:29.732191 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_40e2218a9f58420e9a09ccfbda122485"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.733579 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40e2218a9f58420e9a09ccfbda122485 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.736613 1203 tablet_bootstrap.cc:654] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:29.725091 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 52c05204502a45e18c2b862e21c6f361 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52c05204502a45e18c2b862e21c6f361 already in progress: creating tablet
W20251214 20:52:29.738372 454 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d0f7bd831b4474a90288ac7fa600436 consensus state has no leader
W20251214 20:52:29.737937 991 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 72054us
I20251214 20:52:29.740358 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 57f2314d33fa474ca36dafe0a0eae5ae with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82068013e8471c9309bcf8e8604ece) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:29.740422 2604 raft_consensus.cc:493] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.740603 2604 raft_consensus.cc:515] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.741232 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e11723c042a4637839f1b13bed0935a with delete_type TABLET_DATA_DELETED (replaced by tablet 6e005feeb95b48b9ad33612ab81d5bce) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.741304 2604 leader_election.cc:290] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:29.741496 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a57409d4f54f17be792e6f3f17d7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.741519 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.741809 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet f98be57ce1b14f20a3afcb79979b6c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.742094 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet f7f06eded6c340c58101fcc644ebcf50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:29.742450 713 consensus_peers.cc:597] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:29.743531 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a23561b9f4e48bbbd796e14fafec086" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:29.744014 1206 tablet_bootstrap.cc:492] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
W20251214 20:52:29.744131 989 leader_election.cc:343] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:29.744221 1206 ts_tablet_manager.cc:1403] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.491s user 0.004s sys 0.003s
I20251214 20:52:29.745210 1206 raft_consensus.cc:359] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.745448 1206 raft_consensus.cc:385] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.745536 1206 raft_consensus.cc:740] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.745548 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 98b7cac3930a4a9eb16ac413c8f1e092 (table default.Table_e5f3db562155448e91e2e4314254cbe6 [id=826a2666d167435c8fdfd9ab8b8819e6]) successfully deleted
I20251214 20:52:29.745823 1206 consensus_queue.cc:260] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:29.745898 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a0976ffcc649461fac74e20b5e402275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0976ffcc649461fac74e20b5e402275 already in progress: creating tablet
I20251214 20:52:29.747392 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 18e119bc44874aecb4e83c8d6040c8cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.747857 745 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a57409d4f54f17be792e6f3f17d7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:29.749958 2564 raft_consensus.cc:493] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.750119 2564 raft_consensus.cc:515] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:29.751045 1206 ts_tablet_manager.cc:1434] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251214 20:52:29.751742 1856 ts_tablet_manager.cc:1929] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:29.751858 1856 log.cc:1199] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/d13f93bb5e5f4b59bcf9f4bfed070f25
I20251214 20:52:29.752228 1856 ts_tablet_manager.cc:1950] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:29.755050 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a133da1dda5b4d66b144f6b1fe90f605" } new_table_name: "default.Table_7fbc660486544d9fb711dcdca0660237" modify_external_catalogs: true
I20251214 20:52:29.757776 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f465cef46cc4e23982dced0d2ea9e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.758052 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 18e119bc44874aecb4e83c8d6040c8cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.758404 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_e7a205677f614917bc69ca73b5fb54da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:29.758673 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 471) took 1592 ms. Trace:
I20251214 20:52:29.758752 1856 rpcz_store.cc:276] 1214 20:52:28.166523 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.176926 (+ 10403us) service_pool.cc:225] Handling call
1214 20:52:29.700763 (+1523837us) ts_tablet_manager.cc:1151] Deleting tablet d13f93bb5e5f4b59bcf9f4bfed070f25
1214 20:52:29.700779 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.731470 (+ 30691us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.751587 (+ 20117us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.758651 (+ 7064us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5117,"tablet-delete.queue_time_us":1523575}
I20251214 20:52:29.759097 1856 tablet_replica.cc:333] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:29.759172 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7a205677f614917bc69ca73b5fb54da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.759346 1856 raft_consensus.cc:2243] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:29.759500 1856 raft_consensus.cc:2272] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.763602 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_62fb3a07f4a04029a6080d9d0e35d151"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.764511 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62fb3a07f4a04029a6080d9d0e35d151 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.760068 1856 ts_tablet_manager.cc:1916] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.766409 880 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a57409d4f54f17be792e6f3f17d7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.774706 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "492770a7b8b74ba49315f974cc54eedb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:29.777333 1977 ts_tablet_manager.cc:1929] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:29.777503 1977 log.cc:1199] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/1e31615d1277412eb8bd80451f2c4366
I20251214 20:52:29.777935 1977 ts_tablet_manager.cc:1950] T 1e31615d1277412eb8bd80451f2c4366 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:29.778327 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 391) took 2339 ms. Trace:
I20251214 20:52:29.778406 1977 rpcz_store.cc:276] 1214 20:52:27.438811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.439410 (+ 599us) service_pool.cc:225] Handling call
1214 20:52:29.612413 (+2173003us) ts_tablet_manager.cc:1151] Deleting tablet 1e31615d1277412eb8bd80451f2c4366
1214 20:52:29.612422 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.627615 (+ 15193us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.777187 (+149572us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.778306 (+ 1119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2172869}
I20251214 20:52:29.778903 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 393) took 2204 ms. Trace:
I20251214 20:52:29.778966 1977 rpcz_store.cc:276] 1214 20:52:27.574603 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.574671 (+ 68us) service_pool.cc:225] Handling call
1214 20:52:29.778761 (+2204090us) ts_tablet_manager.cc:1151] Deleting tablet 1c5ee4114cb34540af2c5b332ebe82ab
1214 20:52:29.778770 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.778888 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2203863}
I20251214 20:52:29.784231 885 tablet_service.cc:1552] Processing DeleteTablet for tablet f7f06eded6c340c58101fcc644ebcf50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.785120 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 395) took 2175 ms. Trace:
I20251214 20:52:29.785214 1977 rpcz_store.cc:276] 1214 20:52:27.610077 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.610274 (+ 197us) service_pool.cc:225] Handling call
1214 20:52:29.784950 (+2174676us) ts_tablet_manager.cc:1151] Deleting tablet 6d535d8f71b44c81bbba6937e804d900
1214 20:52:29.784966 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.785098 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2174453}
I20251214 20:52:29.785532 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 396) took 2175 ms. Trace:
I20251214 20:52:29.785470 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 1e31615d1277412eb8bd80451f2c4366 (table default.Table_246962600e464910aa3a756037dd08e9 [id=cd14def069aa430994cd391ddab6c4dd]) successfully deleted
I20251214 20:52:29.785602 1977 rpcz_store.cc:276] 1214 20:52:27.610203 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.610516 (+ 313us) service_pool.cc:225] Handling call
1214 20:52:29.785443 (+2174927us) ts_tablet_manager.cc:1151] Deleting tablet 9e1835374cfa48939d72476870accc3b
1214 20:52:29.785451 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.785520 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2174784}
I20251214 20:52:29.785842 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 399) took 2133 ms. Trace:
I20251214 20:52:29.785898 1977 rpcz_store.cc:276] 1214 20:52:27.652128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.652211 (+ 83us) service_pool.cc:225] Handling call
1214 20:52:29.785765 (+2133554us) ts_tablet_manager.cc:1151] Deleting tablet ed9790fbebda4f848291930bf1dd8174
1214 20:52:29.785772 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.785831 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2132630}
I20251214 20:52:29.786134 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 401) took 2103 ms. Trace:
I20251214 20:52:29.787384 884 tablet_service.cc:1552] Processing DeleteTablet for tablet f98be57ce1b14f20a3afcb79979b6c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.788959 1856 ts_tablet_manager.cc:1929] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:29.789482 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 18e119bc44874aecb4e83c8d6040c8cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:29.786202 1977 rpcz_store.cc:276] 1214 20:52:27.682690 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.686801 (+ 4111us) service_pool.cc:225] Handling call
1214 20:52:29.786057 (+2099256us) ts_tablet_manager.cc:1151] Deleting tablet 52c05204502a45e18c2b862e21c6f361
1214 20:52:29.786065 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.786122 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2093252}
I20251214 20:52:29.790031 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.790422 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 406) took 1994 ms. Trace:
I20251214 20:52:29.790555 1977 rpcz_store.cc:276] 1214 20:52:27.795708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.797770 (+ 2062us) service_pool.cc:225] Handling call
1214 20:52:29.790225 (+1992455us) ts_tablet_manager.cc:1151] Deleting tablet 2f9def8e4fd449599820a63845397750
1214 20:52:29.790235 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.790381 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1992088}
I20251214 20:52:29.789176 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.790983 1856 log.cc:1199] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/453cf734f8b34b59aa246d618ffeade5
I20251214 20:52:29.791184 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 407) took 1976 ms. Trace:
I20251214 20:52:29.791359 1856 ts_tablet_manager.cc:1950] T 453cf734f8b34b59aa246d618ffeade5 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:29.791294 1977 rpcz_store.cc:276] 1214 20:52:27.814289 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.819064 (+ 4775us) service_pool.cc:225] Handling call
1214 20:52:29.790904 (+1971840us) ts_tablet_manager.cc:1151] Deleting tablet d171301941fa43908e05984c3b016007
1214 20:52:29.790912 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.791166 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1953084}
I20251214 20:52:29.791697 1977 tablet_replica.cc:333] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:29.791657 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 470) took 1625 ms. Trace:
I20251214 20:52:29.792189 1977 raft_consensus.cc:2243] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:29.792189 2564 leader_election.cc:290] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:29.791740 1856 rpcz_store.cc:276] 1214 20:52:28.166331 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.182389 (+ 16058us) service_pool.cc:225] Handling call
1214 20:52:29.759048 (+1576659us) ts_tablet_manager.cc:1151] Deleting tablet 453cf734f8b34b59aa246d618ffeade5
1214 20:52:29.759057 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.769998 (+ 10941us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.788808 (+ 18810us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.791641 (+ 2833us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576398}
I20251214 20:52:29.794435 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 473) took 1606 ms. Trace:
I20251214 20:52:29.795377 448 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
I20251214 20:52:29.796067 1856 rpcz_store.cc:276] 1214 20:52:28.187618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.193045 (+ 5427us) service_pool.cc:225] Handling call
1214 20:52:29.794302 (+1601257us) ts_tablet_manager.cc:1151] Deleting tablet 0a2fbec0543445b7856ef22c934c6840
1214 20:52:29.794312 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.794417 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1601023}
I20251214 20:52:29.796556 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_68b81e8dee2b457ea98b839dca8e5464" } new_table_name: "default.Table_679d53b1472341ef8612ca53ac1ebda2" modify_external_catalogs: true
I20251214 20:52:29.797573 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 474) took 1527 ms. Trace:
I20251214 20:52:29.798243 1856 rpcz_store.cc:276] 1214 20:52:28.270398 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.271063 (+ 665us) service_pool.cc:225] Handling call
1214 20:52:29.797454 (+1526391us) ts_tablet_manager.cc:1151] Deleting tablet 088c72a6f5014b6fbc6ee060b6b7a590
1214 20:52:29.797463 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.797555 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1526150}
I20251214 20:52:29.786609 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]) successfully deleted
I20251214 20:52:29.799568 1856 tablet_replica.cc:333] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:29.799923 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_93fe3196f8c94960a5c5ee9e1f3923f1 [id=611e3a3e47ba4f91a705e9b8d3c323f3]) successfully deleted
W20251214 20:52:29.800199 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 0a2fbec0543445b7856ef22c934c6840 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2fbec0543445b7856ef22c934c6840 already in progress: creating tablet
W20251214 20:52:29.800460 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 088c72a6f5014b6fbc6ee060b6b7a590 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 088c72a6f5014b6fbc6ee060b6b7a590 already in progress: creating tablet
I20251214 20:52:29.800671 1856 raft_consensus.cc:2243] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:29.801156 1856 raft_consensus.cc:2272] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.802671 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 397) took 2007 ms. Trace:
I20251214 20:52:29.802796 1982 rpcz_store.cc:276] 1214 20:52:27.795216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.797590 (+ 2374us) service_pool.cc:225] Handling call
1214 20:52:29.802491 (+2004901us) ts_tablet_manager.cc:1151] Deleting tablet 2f9def8e4fd449599820a63845397750
1214 20:52:29.802503 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.802642 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2004630}
W20251214 20:52:29.803330 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 2f9def8e4fd449599820a63845397750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f9def8e4fd449599820a63845397750 already in progress: creating tablet
I20251214 20:52:29.803730 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 399) took 1992 ms. Trace:
I20251214 20:52:29.803795 1982 rpcz_store.cc:276] 1214 20:52:27.811552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.811740 (+ 188us) service_pool.cc:225] Handling call
1214 20:52:29.803601 (+1991861us) ts_tablet_manager.cc:1151] Deleting tablet d171301941fa43908e05984c3b016007
1214 20:52:29.803612 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.803712 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1991437}
W20251214 20:52:29.804162 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet d171301941fa43908e05984c3b016007 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d171301941fa43908e05984c3b016007 already in progress: creating tablet
I20251214 20:52:29.804410 1982 tablet_replica.cc:333] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:29.804596 1982 raft_consensus.cc:2243] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:29.804864 1982 raft_consensus.cc:2272] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.805462 1982 ts_tablet_manager.cc:1916] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.806337 1856 ts_tablet_manager.cc:1916] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:29.807617 991 leader_election.cc:343] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:29.808125 991 leader_election.cc:304] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:29.808902 2706 raft_consensus.cc:2749] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:29.793005 1977 raft_consensus.cc:2272] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.810793 2706 consensus_queue.cc:1048] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:29.799149 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e11723c042a4637839f1b13bed0935a with delete_type TABLET_DATA_DELETED (replaced by tablet 6e005feeb95b48b9ad33612ab81d5bce) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:29.815690 1977 ts_tablet_manager.cc:1916] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.806772 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a23561b9f4e48bbbd796e14fafec086" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:29.822072 991 leader_election.cc:343] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:29.822186 991 leader_election.cc:304] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:29.822510 2705 raft_consensus.cc:2749] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:29.828425 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 98b7cac3930a4a9eb16ac413c8f1e092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.828519 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dca369dc5f54a4ea1783e334c5670d6" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:29.832849 447 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d424e3387698434f92e84eb1d43f5949 consensus state has no leader
W20251214 20:52:29.829908 713 consensus_peers.cc:597] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf8643c7e4094f529cd55656c7b3edb6. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:29.849550 1206 tablet_bootstrap.cc:492] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:29.851048 1206 tablet_bootstrap.cc:654] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.834143 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.834208 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e1835374cfa48939d72476870accc3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.853830 1052 raft_consensus.cc:2468] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:29.847831 2604 consensus_queue.cc:1048] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:29.854383 2604 raft_consensus.cc:493] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.854514 2604 raft_consensus.cc:515] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.855057 2604 leader_election.cc:290] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:29.852047 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 492770a7b8b74ba49315f974cc54eedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.856251 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6bdfbe4d15345de8be5de00bfe4ca07" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:29.856791 991 leader_election.cc:343] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:29.859231 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6bdfbe4d15345de8be5de00bfe4ca07" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:29.859723 989 leader_election.cc:343] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:29.859853 989 leader_election.cc:304] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:29.863722 2551 raft_consensus.cc:2749] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:29.866372 1203 tablet_bootstrap.cc:492] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:29.866684 446 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]). Expected version 1 got 0
W20251214 20:52:29.871975 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7885f21a369461c8c80ff18d369e2c6 has no consensus state
I20251214 20:52:29.872434 1203 ts_tablet_manager.cc:1403] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.149s user 0.011s sys 0.016s
I20251214 20:52:29.873229 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:29.876093 2714 raft_consensus.cc:493] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:29.876397 2714 raft_consensus.cc:515] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.876502 450 master_service.cc:954] ReplaceTablet: received request to replace tablet a6df388bdfbc40c183c4b14af2336a6a from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:29.880605 2714 leader_election.cc:290] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:29.880807 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "658e4fe0d188483b9ceb9eb3b3f20835" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:29.882230 1203 raft_consensus.cc:359] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.888561 1203 raft_consensus.cc:385] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.888741 1203 raft_consensus.cc:740] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.886739 1856 ts_tablet_manager.cc:1929] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251214 20:52:29.889195 1856 log.cc:1199] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/956e4b4493f24109ab1251dde0a4cfc0
I20251214 20:52:29.890341 1856 ts_tablet_manager.cc:1950] T 956e4b4493f24109ab1251dde0a4cfc0 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:29.889736 1203 consensus_queue.cc:260] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.893074 1202 tablet_bootstrap.cc:492] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:29.893173 851 leader_election.cc:304] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:29.893257 1202 ts_tablet_manager.cc:1403] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.436s user 0.011s sys 0.009s
I20251214 20:52:29.895288 1203 ts_tablet_manager.cc:1434] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.023s user 0.002s sys 0.001s
I20251214 20:52:29.895841 1203 tablet_bootstrap.cc:492] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:29.898388 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_47862fea18504c61af1783e5d8fca95e" } modify_external_catalogs: true
I20251214 20:52:29.898629 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_47862fea18504c61af1783e5d8fca95e" } modify_external_catalogs: true
I20251214 20:52:29.900103 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_af66f4d0779b46a5a91d7d2490a845c9" } modify_external_catalogs: true
I20251214 20:52:29.900322 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_af66f4d0779b46a5a91d7d2490a845c9" } modify_external_catalogs: true
I20251214 20:52:29.902947 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 475) took 1534 ms. Trace:
I20251214 20:52:29.903102 1856 rpcz_store.cc:276] 1214 20:52:28.368389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.372683 (+ 4294us) service_pool.cc:225] Handling call
1214 20:52:29.799515 (+1426832us) ts_tablet_manager.cc:1151] Deleting tablet 956e4b4493f24109ab1251dde0a4cfc0
1214 20:52:29.799526 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.815737 (+ 16211us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.886582 (+ 70845us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.902923 (+ 16341us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6845,"tablet-delete.queue_time_us":1425965}
I20251214 20:52:29.903246 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet b1f4ea72c6274569a98cab6afdc55d55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.903645 1856 tablet_replica.cc:333] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:29.904017 1203 tablet_bootstrap.cc:654] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.904321 446 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5e8a4451754247c2add9a3245b63b050" } modify_external_catalogs: true
I20251214 20:52:29.904511 446 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_5e8a4451754247c2add9a3245b63b050" } modify_external_catalogs: true
I20251214 20:52:29.906122 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_35def73a244b4885ad22656a1df2b697" } modify_external_catalogs: true
I20251214 20:52:29.906358 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_35def73a244b4885ad22656a1df2b697" } modify_external_catalogs: true
I20251214 20:52:29.907452 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]) successfully deleted
I20251214 20:52:29.904165 1856 raft_consensus.cc:2243] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:29.908716 1856 raft_consensus.cc:2272] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.909592 1856 ts_tablet_manager.cc:1916] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.909897 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3377d29944b3480a88fd7c603ccd0a02" } modify_external_catalogs: true
I20251214 20:52:29.910164 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_3377d29944b3480a88fd7c603ccd0a02" } modify_external_catalogs: true
I20251214 20:52:29.912227 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_cca1629ea2974eddb3a2fab0ae9f4500"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:29.912325 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_a54f22962823421ea5ab3033e0d9add0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.912847 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cca1629ea2974eddb3a2fab0ae9f4500 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:29.913008 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a54f22962823421ea5ab3033e0d9add0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.922286 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6085c994e5404569b349fe215f389970" } new_table_name: "default.Table_9af6d4105a984471b526cd9d4d0941f8" modify_external_catalogs: true
I20251214 20:52:29.922509 1856 ts_tablet_manager.cc:1929] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251214 20:52:29.922647 1856 log.cc:1199] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/b70579ddf96f4499b4e4f235e8f69d67
I20251214 20:52:29.922987 1856 ts_tablet_manager.cc:1950] T b70579ddf96f4499b4e4f235e8f69d67 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:29.923893 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 476) took 1510 ms. Trace:
I20251214 20:52:29.924075 1856 rpcz_store.cc:276] 1214 20:52:28.413318 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.413367 (+ 49us) service_pool.cc:225] Handling call
1214 20:52:29.903579 (+1490212us) ts_tablet_manager.cc:1151] Deleting tablet b70579ddf96f4499b4e4f235e8f69d67
1214 20:52:29.903599 (+ 20us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.919413 (+ 15814us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.922390 (+ 2977us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.923870 (+ 1480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1490013}
I20251214 20:52:29.924764 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_3a068835084e406eb4047a827faeb5d7 [id=bdb017130eed4d76b635603560c91d41]) successfully deleted
I20251214 20:52:29.925181 1856 tablet_replica.cc:333] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:29.928646 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4643effa71d4d18850913a59ae1c75d consensus state has no leader
I20251214 20:52:29.929581 2571 raft_consensus.cc:515] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:29.929894 1202 raft_consensus.cc:359] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.930074 1202 raft_consensus.cc:385] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.930127 1202 raft_consensus.cc:740] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.930228 2571 leader_election.cc:290] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:29.930344 1202 consensus_queue.cc:260] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:29.930902 2571 raft_consensus.cc:2804] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:29.931001 2571 raft_consensus.cc:493] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:29.931074 2571 raft_consensus.cc:3060] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:29.934176 1977 ts_tablet_manager.cc:1929] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:29.934299 1977 log.cc:1199] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/17550eb0ee6b476f8a8c304451f678b2
I20251214 20:52:29.936808 1203 tablet_bootstrap.cc:492] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:29.936987 1203 ts_tablet_manager.cc:1403] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20251214 20:52:29.937898 1203 raft_consensus.cc:359] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:29.938081 1203 raft_consensus.cc:385] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:29.938128 1203 raft_consensus.cc:740] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:29.938326 1203 consensus_queue.cc:260] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:29.939500 1203 ts_tablet_manager.cc:1434] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:29.940769 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_fefd2aebe38641a6892916b197d27abc" } new_table_name: "default.Table_cddd59a230e04595bffdf9e0a0ed1471" modify_external_catalogs: true
I20251214 20:52:29.946228 1032 tablet_service.cc:1505] Processing CreateTablet for tablet c7885f21a369461c8c80ff18d369e2c6 (DEFAULT_TABLE table=default.Table_7fbc660486544d9fb711dcdca0660237 [id=a20045f442604c45a67b9b06f1eb4610]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.947525 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 4106fb80b4304735bc3faba7a7bf342f (DEFAULT_TABLE table=default.Table_b567effdb6834f7b87ce4aba9e555913 [id=c216b38d4fce484da23d096493ac2934]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.947685 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 6bf7317683ff4a52b3503687f4131ad3 (DEFAULT_TABLE table=default.Table_6588026e2d974db9b72d2b81ce8310cb [id=483610ffc7db4e848d0756374015974f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.949877 1202 ts_tablet_manager.cc:1434] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.057s user 0.004s sys 0.000s
I20251214 20:52:29.950277 1202 tablet_bootstrap.cc:492] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:29.951105 1977 ts_tablet_manager.cc:1950] T 17550eb0ee6b476f8a8c304451f678b2 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:29.951335 2571 raft_consensus.cc:515] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:29.951519 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 412) took 2052 ms. Trace:
I20251214 20:52:29.951617 1977 rpcz_store.cc:276] 1214 20:52:27.898967 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.899031 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:29.791625 (+1892594us) ts_tablet_manager.cc:1151] Deleting tablet 17550eb0ee6b476f8a8c304451f678b2
1214 20:52:29.791635 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.819003 (+ 27368us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.934046 (+115043us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:29.951496 (+ 17450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1892332}
I20251214 20:52:29.951927 1977 tablet_replica.cc:333] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:29.952049 2571 leader_election.cc:290] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:29.952097 1977 raft_consensus.cc:2243] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:29.952406 1977 pending_rounds.cc:70] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:29.952450 1977 pending_rounds.cc:77] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493763222249472 op_type: NO_OP noop_request { }
I20251214 20:52:29.952535 1977 raft_consensus.cc:2889] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:29.952631 1977 raft_consensus.cc:2272] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:29.953235 1977 ts_tablet_manager.cc:1916] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:29.953392 1023 tablet_service.cc:1505] Processing CreateTablet for tablet d48c6789e5724e989900d3351ecde3ab (DEFAULT_TABLE table=default.Table_b6011f3e00584819944d165d653d06ac [id=103545908ce040858f00aed970b27479]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.953454 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 83da479367fe4b949f55bc6756883892 (DEFAULT_TABLE table=default.Table_f7d781182d07441a824d887e07286dc3 [id=6170186dc9cc4f4e93394530eb7c3a56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.953859 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 6cd04f871edb4fb78d6ec852c062ef54 (DEFAULT_TABLE table=default.Table_6fedc617de7e4cd985c616534bda854a [id=5c2a10eb6dca403a94f387331ec8c3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.955554 1025 tablet_service.cc:1505] Processing CreateTablet for tablet e1dbaa39efa04086a22cc3908ed691d4 (DEFAULT_TABLE table=default.Table_8f475bc0f2dd4602b5f8e057241c3c16 [id=4390cb960d564bb5bc7e0cefa5bf3004]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.957041 1021 tablet_service.cc:1505] Processing CreateTablet for tablet de9c830cf9c74cff88cb6f42cbb5f7cd (DEFAULT_TABLE table=default.Table_f9499a2114f44353aaf65b35ed56f119 [id=7dd81c38dde94205a84d6f7841d2bb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.957862 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 556ac6b97fab4ea5a43089c0f96c38e5 (DEFAULT_TABLE table=default.Table_8f5d31355346492989caa148a8f1440b [id=ec2fd20884ff4bdca7be9d32d3bfe9ae]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:29.935632 904 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.003s sys 0.014s
W20251214 20:52:29.958542 904 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s user 0.003s sys 0.014s
I20251214 20:52:29.960477 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 780be07e6f5d425d9605402b6db63012 (DEFAULT_TABLE table=default.Table_ee7a207364d74e6b81d6a72e57cacbd5 [id=b251a30885f6435fbfa430b6b50aee7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.962153 1029 tablet_service.cc:1505] Processing CreateTablet for tablet ae54e7a5104e4cd588a254551fc52477 (DEFAULT_TABLE table=default.Table_68c79df4e9e0446f81c8500bcfa423a6 [id=7e9bbbe22677488d9509dc201b2d7234]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.965442 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 1ee28c4ac91448629c0bddb3143f7df0 (DEFAULT_TABLE table=default.Table_bf1d2eb618b64a4e86cd121824fec0cb [id=1c14038af47c43c28ec8370fba0a14df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.965442 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 19008711a4754ca7b38a3ef2f0cdbf1d (DEFAULT_TABLE table=default.Table_a07fe706644d4c63b966f0751b757642 [id=8845cef019b44676a83a3173c5df0bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.968469 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet bf37c50e511f454ca4887c7f6ca1a746
I20251214 20:52:29.969141 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet bf37c50e511f454ca4887c7f6ca1a746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:29.969440 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_8923490b824a47139d8f739d65e4845b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:29.970036 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8923490b824a47139d8f739d65e4845b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:29.983915 1202 tablet_bootstrap.cc:654] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.985538 1203 tablet_bootstrap.cc:492] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:29.988871 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 5afb7fb4f0a44b0092fbef5bbec9f396 (DEFAULT_TABLE table=default.Table_87f9c82a6a89490391b5c78001639d2b [id=8203de75aba94c3f8561f18adaaa825e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.989140 1203 tablet_bootstrap.cc:654] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:29.989331 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afb7fb4f0a44b0092fbef5bbec9f396. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.992173 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 69f9d5838e544002ab49f0ce21b6f7ef (DEFAULT_TABLE table=default.Table_9ff099dcfac14636a9dcaa70207c86a6 [id=4a1816b57ff84e459394e59b683d8ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.992524 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f9d5838e544002ab49f0ce21b6f7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:29.995396 746 tablet_service.cc:1505] Processing CreateTablet for tablet 6cd04f871edb4fb78d6ec852c062ef54 (DEFAULT_TABLE table=default.Table_6fedc617de7e4cd985c616534bda854a [id=5c2a10eb6dca403a94f387331ec8c3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.997676 747 tablet_service.cc:1505] Processing CreateTablet for tablet 4106fb80b4304735bc3faba7a7bf342f (DEFAULT_TABLE table=default.Table_b567effdb6834f7b87ce4aba9e555913 [id=c216b38d4fce484da23d096493ac2934]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.998387 745 tablet_service.cc:1505] Processing CreateTablet for tablet 6bf7317683ff4a52b3503687f4131ad3 (DEFAULT_TABLE table=default.Table_6588026e2d974db9b72d2b81ce8310cb [id=483610ffc7db4e848d0756374015974f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.998543 749 tablet_service.cc:1505] Processing CreateTablet for tablet e1dbaa39efa04086a22cc3908ed691d4 (DEFAULT_TABLE table=default.Table_8f475bc0f2dd4602b5f8e057241c3c16 [id=4390cb960d564bb5bc7e0cefa5bf3004]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.999749 752 tablet_service.cc:1505] Processing CreateTablet for tablet d48c6789e5724e989900d3351ecde3ab (DEFAULT_TABLE table=default.Table_b6011f3e00584819944d165d653d06ac [id=103545908ce040858f00aed970b27479]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.000944 753 tablet_service.cc:1505] Processing CreateTablet for tablet 83da479367fe4b949f55bc6756883892 (DEFAULT_TABLE table=default.Table_f7d781182d07441a824d887e07286dc3 [id=6170186dc9cc4f4e93394530eb7c3a56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.001260 742 tablet_service.cc:1505] Processing CreateTablet for tablet de9c830cf9c74cff88cb6f42cbb5f7cd (DEFAULT_TABLE table=default.Table_f9499a2114f44353aaf65b35ed56f119 [id=7dd81c38dde94205a84d6f7841d2bb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:29.993628 743 tablet_service.cc:1505] Processing CreateTablet for tablet c7885f21a369461c8c80ff18d369e2c6 (DEFAULT_TABLE table=default.Table_7fbc660486544d9fb711dcdca0660237 [id=a20045f442604c45a67b9b06f1eb4610]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.002257 750 tablet_service.cc:1505] Processing CreateTablet for tablet ae54e7a5104e4cd588a254551fc52477 (DEFAULT_TABLE table=default.Table_68c79df4e9e0446f81c8500bcfa423a6 [id=7e9bbbe22677488d9509dc201b2d7234]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.002581 744 tablet_service.cc:1505] Processing CreateTablet for tablet 19008711a4754ca7b38a3ef2f0cdbf1d (DEFAULT_TABLE table=default.Table_a07fe706644d4c63b966f0751b757642 [id=8845cef019b44676a83a3173c5df0bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.002913 741 tablet_service.cc:1505] Processing CreateTablet for tablet 556ac6b97fab4ea5a43089c0f96c38e5 (DEFAULT_TABLE table=default.Table_8f5d31355346492989caa148a8f1440b [id=ec2fd20884ff4bdca7be9d32d3bfe9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.003258 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4106fb80b4304735bc3faba7a7bf342f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.003515 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19008711a4754ca7b38a3ef2f0cdbf1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.003793 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83da479367fe4b949f55bc6756883892. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.003921 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1dbaa39efa04086a22cc3908ed691d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.004349 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7885f21a369461c8c80ff18d369e2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.004570 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48c6789e5724e989900d3351ecde3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.004741 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bf7317683ff4a52b3503687f4131ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.005245 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83da479367fe4b949f55bc6756883892. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.008442 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de9c830cf9c74cff88cb6f42cbb5f7cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:30.010221 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 117059us
I20251214 20:52:30.010299 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 780be07e6f5d425d9605402b6db63012. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.011041 1977 ts_tablet_manager.cc:1929] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:30.011334 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 556ac6b97fab4ea5a43089c0f96c38e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.011739 756 tablet_service.cc:1505] Processing CreateTablet for tablet 780be07e6f5d425d9605402b6db63012 (DEFAULT_TABLE table=default.Table_ee7a207364d74e6b81d6a72e57cacbd5 [id=b251a30885f6435fbfa430b6b50aee7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.015390 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19008711a4754ca7b38a3ef2f0cdbf1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.015591 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ee28c4ac91448629c0bddb3143f7df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.015777 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1dbaa39efa04086a22cc3908ed691d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.016906 755 tablet_service.cc:1552] Processing DeleteTablet for tablet bf37c50e511f454ca4887c7f6ca1a746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:30.018465 754 tablet_service.cc:1505] Processing CreateTablet for tablet 700ed62ef2e84719a7a8e9879feaf45c (DEFAULT_TABLE table=default.Table_65d3039202a645e899e5f752e1e3db42 [id=b6bebe3ab7e2449dafc61fdda3e3c38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.019218 2551 raft_consensus.cc:493] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.019362 2551 raft_consensus.cc:515] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.019570 748 tablet_service.cc:1505] Processing CreateTablet for tablet 1ee28c4ac91448629c0bddb3143f7df0 (DEFAULT_TABLE table=default.Table_bf1d2eb618b64a4e86cd121824fec0cb [id=1c14038af47c43c28ec8370fba0a14df]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:30.019812 989 consensus_peers.cc:597] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:30.019949 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae54e7a5104e4cd588a254551fc52477. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.020124 2551 leader_election.cc:290] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:30.020164 989 consensus_peers.cc:597] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:30.020768 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4106fb80b4304735bc3faba7a7bf342f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.023191 751 tablet_service.cc:1505] Processing CreateTablet for tablet 5afb7fb4f0a44b0092fbef5bbec9f396 (DEFAULT_TABLE table=default.Table_87f9c82a6a89490391b5c78001639d2b [id=8203de75aba94c3f8561f18adaaa825e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.023561 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25dcf491653147e7831217c1ec7a2e0c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:30.023589 755 tablet_service.cc:1505] Processing CreateTablet for tablet 69f9d5838e544002ab49f0ce21b6f7ef (DEFAULT_TABLE table=default.Table_9ff099dcfac14636a9dcaa70207c86a6 [id=4a1816b57ff84e459394e59b683d8ec6]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:30.024626 989 leader_election.cc:343] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.025884 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cd04f871edb4fb78d6ec852c062ef54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.027623 2714 raft_consensus.cc:493] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.027771 2714 raft_consensus.cc:515] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.028455 2714 leader_election.cc:290] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:30.031916 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738668adf2874d6894db6fa875f6990c" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:30.034690 1977 log.cc:1199] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/0c973e0111804b8b869068bdcc5869bf
I20251214 20:52:30.035373 1977 ts_tablet_manager.cc:1950] T 0c973e0111804b8b869068bdcc5869bf P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:30.036345 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 415) took 2136 ms. Trace:
I20251214 20:52:30.036450 1977 rpcz_store.cc:276] 1214 20:52:27.899857 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.899957 (+ 100us) service_pool.cc:225] Handling call
1214 20:52:29.951877 (+2051920us) ts_tablet_manager.cc:1151] Deleting tablet 0c973e0111804b8b869068bdcc5869bf
1214 20:52:29.951885 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.990880 (+ 38995us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.010904 (+ 20024us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.036320 (+ 25416us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2051215}
I20251214 20:52:30.036809 1977 tablet_replica.cc:333] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:30.037014 1977 raft_consensus.cc:2243] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.037176 1977 raft_consensus.cc:2272] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.037742 1977 ts_tablet_manager.cc:1916] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.040565 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "876d2a1f1d1c4f6882a5857b43568ee5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:30.040789 1050 raft_consensus.cc:3060] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:30.040629 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 279918us
I20251214 20:52:30.042127 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dca369dc5f54a4ea1783e334c5670d6" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:30.042862 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "876d2a1f1d1c4f6882a5857b43568ee5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:30.043939 1050 raft_consensus.cc:2468] T 876d2a1f1d1c4f6882a5857b43568ee5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:30.044512 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "658e4fe0d188483b9ceb9eb3b3f20835" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:30.045614 2717 tablet.cc:1722] T d9f198ef4dfa4f5791144b9a24fe693f P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:30.050057 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dca369dc5f54a4ea1783e334c5670d6" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:30.051899 848 leader_election.cc:343] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.052079 1206 tablet_bootstrap.cc:492] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:30.052423 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 556ac6b97fab4ea5a43089c0f96c38e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.052680 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f9d5838e544002ab49f0ce21b6f7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.052901 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 700ed62ef2e84719a7a8e9879feaf45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.053457 1206 ts_tablet_manager.cc:1403] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.207s user 0.009s sys 0.001s
W20251214 20:52:30.053562 848 leader_election.cc:343] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:30.053787 848 consensus_peers.cc:597] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:30.053977 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 780be07e6f5d425d9605402b6db63012. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.054314 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afb7fb4f0a44b0092fbef5bbec9f396. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.054342 1206 raft_consensus.cc:359] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.054476 1206 raft_consensus.cc:385] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.054523 1206 raft_consensus.cc:740] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.054687 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae54e7a5104e4cd588a254551fc52477. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.054836 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de9c830cf9c74cff88cb6f42cbb5f7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.054955 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48c6789e5724e989900d3351ecde3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.055083 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ee28c4ac91448629c0bddb3143f7df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.055449 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7885f21a369461c8c80ff18d369e2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.055823 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bf7317683ff4a52b3503687f4131ad3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:30.057866 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1def5f0de5ce4eb0bed36f626c4f0683 consensus state has no leader
W20251214 20:52:30.058902 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 202b1d53cab644c98838b0ee81f29f22 consensus state has no leader
I20251214 20:52:30.059238 453 master_service.cc:954] ReplaceTablet: received request to replace tablet f84e9dfc23d34eb0a379e8ac5a442e59 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:30.061352 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cd04f871edb4fb78d6ec852c062ef54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.062240 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dca369dc5f54a4ea1783e334c5670d6" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:30.062441 1052 raft_consensus.cc:3060] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:30.065522 1052 raft_consensus.cc:2468] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:30.068332 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 700ed62ef2e84719a7a8e9879feaf45c (DEFAULT_TABLE table=default.Table_65d3039202a645e899e5f752e1e3db42 [id=b6bebe3ab7e2449dafc61fdda3e3c38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.068917 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 700ed62ef2e84719a7a8e9879feaf45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.070098 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738668adf2874d6894db6fa875f6990c" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:30.071144 851 leader_election.cc:304] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:30.072113 2712 raft_consensus.cc:2804] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:30.072213 2712 raft_consensus.cc:697] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:30.072448 2712 consensus_queue.cc:237] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.071694 2725 tablet.cc:1722] T 738668adf2874d6894db6fa875f6990c P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:30.079470 2721 consensus_queue.cc:1048] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:30.080253 851 leader_election.cc:304] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:30.080921 1033 tablet_service.cc:1505] Processing CreateTablet for tablet fd3a49d6acae40d099cdd99981ce9d46 (DEFAULT_TABLE table=default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.081532 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd3a49d6acae40d099cdd99981ce9d46. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:30.081487 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 6d535d8f71b44c81bbba6937e804d900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d535d8f71b44c81bbba6937e804d900 already in progress: creating tablet
W20251214 20:52:30.081938 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 9e1835374cfa48939d72476870accc3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e1835374cfa48939d72476870accc3b already in progress: creating tablet
I20251214 20:52:30.082193 2656 raft_consensus.cc:2804] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:30.082350 2656 raft_consensus.cc:697] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
W20251214 20:52:30.082433 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ed9790fbebda4f848291930bf1dd8174 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed9790fbebda4f848291930bf1dd8174 already in progress: creating tablet
I20251214 20:52:30.082679 2656 consensus_queue.cc:237] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:30.083323 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 52c05204502a45e18c2b862e21c6f361 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52c05204502a45e18c2b862e21c6f361 already in progress: creating tablet
I20251214 20:52:30.085647 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25dcf491653147e7831217c1ec7a2e0c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:30.086513 991 leader_election.cc:343] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.086673 991 leader_election.cc:304] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:30.087687 2705 raft_consensus.cc:2749] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:30.089718 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2f9def8e4fd449599820a63845397750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f9def8e4fd449599820a63845397750 already in progress: creating tablet
W20251214 20:52:30.090176 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet d171301941fa43908e05984c3b016007 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d171301941fa43908e05984c3b016007 already in progress: creating tablet
I20251214 20:52:30.091512 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 17550eb0ee6b476f8a8c304451f678b2 (table default.Table_6f66a268b27e4f9b8d09597224149553 [id=d36ff09def9741f8b440f7bdbfdc21a8]) successfully deleted
I20251214 20:52:30.091889 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 0c973e0111804b8b869068bdcc5869bf (table default.Table_d8953672be144941ab3319e5477b8c57 [id=8f35dde4f9204c799e62c4d656abfd0f]) successfully deleted
I20251214 20:52:30.092671 881 tablet_service.cc:1505] Processing CreateTablet for tablet 6cd04f871edb4fb78d6ec852c062ef54 (DEFAULT_TABLE table=default.Table_6fedc617de7e4cd985c616534bda854a [id=5c2a10eb6dca403a94f387331ec8c3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.093083 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cd04f871edb4fb78d6ec852c062ef54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.094971 1021 tablet_service.cc:1505] Processing CreateTablet for tablet c5478cba0bf24f8c969fd1b0998b9db3 (DEFAULT_TABLE table=default.Table_0f409000b9514206a08757040e81a6a9 [id=5692c4a553044543ab275cff0f7a0566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.095450 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5478cba0bf24f8c969fd1b0998b9db3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:30.095453 848 leader_election.cc:343] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.099886 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 2ea3c1e1ca5a41df8a01546bc08c3e23 (DEFAULT_TABLE table=default.Table_80391b5a5aa9425794dfb7997c659009 [id=893edd42fa04409b9d8e4a4e46d38ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.103226 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea3c1e1ca5a41df8a01546bc08c3e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.104705 1030 tablet_service.cc:1505] Processing CreateTablet for tablet d39e245bc5a04c1186d0973645aa1b54 (DEFAULT_TABLE table=default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.105193 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d39e245bc5a04c1186d0973645aa1b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.109112 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 86ede5e41a03485082f4531fc42dfe56 (DEFAULT_TABLE table=default.Table_ee3c57e398fd468fa1e4e7cc22f6747a [id=ce8357dfd1024446a69a3ee196ab4b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.109669 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ede5e41a03485082f4531fc42dfe56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.114213 742 tablet_service.cc:1505] Processing CreateTablet for tablet d39e245bc5a04c1186d0973645aa1b54 (DEFAULT_TABLE table=default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.114595 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d39e245bc5a04c1186d0973645aa1b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.115504 744 tablet_service.cc:1505] Processing CreateTablet for tablet fd3a49d6acae40d099cdd99981ce9d46 (DEFAULT_TABLE table=default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.115803 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd3a49d6acae40d099cdd99981ce9d46. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:30.120816 848 consensus_peers.cc:597] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:30.126943 757 tablet_service.cc:1505] Processing CreateTablet for tablet c5478cba0bf24f8c969fd1b0998b9db3 (DEFAULT_TABLE table=default.Table_0f409000b9514206a08757040e81a6a9 [id=5692c4a553044543ab275cff0f7a0566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.127460 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5478cba0bf24f8c969fd1b0998b9db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.133591 712 leader_election.cc:304] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:30.136310 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 5f251ff9b3fa4fe8ababab456733bcdd (DEFAULT_TABLE table=default.Table_b9ef85d4768d42d28d635a7c3eac5f4c [id=7be3dc7e56824beb9f13c242645bdff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.136766 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f251ff9b3fa4fe8ababab456733bcdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.136687 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 6404321f391944769986905a12fbab1c (DEFAULT_TABLE table=default.Table_4abdf07a08814d65acc20b6c32bfd415 [id=aeb3160d155949699e454f389fd07aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.136809 1856 raft_consensus.cc:2243] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.136991 1856 raft_consensus.cc:2272] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.137130 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6404321f391944769986905a12fbab1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.137323 1024 tablet_service.cc:1505] Processing CreateTablet for tablet f6d51d07f7e9450a844be9afddb60d03 (DEFAULT_TABLE table=default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.137569 1856 ts_tablet_manager.cc:1916] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.137632 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d51d07f7e9450a844be9afddb60d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.141789 1982 ts_tablet_manager.cc:1929] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.141971 1982 log.cc:1199] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/2accf8a160584c82bb695a52e3f2b05c
I20251214 20:52:30.143676 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 3373921f73b64fae954ea7c5d5f1d93f (DEFAULT_TABLE table=default.Table_5cda3bedf2d24fc2b279c049146aa2c5 [id=10ef2e196edd432aa21fd7d31fa4366f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.144095 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3373921f73b64fae954ea7c5d5f1d93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.145782 756 tablet_service.cc:1505] Processing CreateTablet for tablet 86ede5e41a03485082f4531fc42dfe56 (DEFAULT_TABLE table=default.Table_ee3c57e398fd468fa1e4e7cc22f6747a [id=ce8357dfd1024446a69a3ee196ab4b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.146154 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ede5e41a03485082f4531fc42dfe56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.147691 1203 tablet_bootstrap.cc:492] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:30.148044 1203 ts_tablet_manager.cc:1403] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.163s user 0.006s sys 0.003s
I20251214 20:52:30.148869 1203 raft_consensus.cc:359] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.149142 1203 raft_consensus.cc:385] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.149227 1203 raft_consensus.cc:740] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.149545 1203 consensus_queue.cc:260] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.150185 1203 ts_tablet_manager.cc:1434] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:30.150514 1203 tablet_bootstrap.cc:492] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:30.152187 1202 tablet_bootstrap.cc:492] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:30.152417 1202 ts_tablet_manager.cc:1403] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.202s user 0.011s sys 0.003s
I20251214 20:52:30.153280 1202 raft_consensus.cc:359] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.153517 1202 raft_consensus.cc:385] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.153601 1202 raft_consensus.cc:740] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.153834 1202 consensus_queue.cc:260] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.154034 744 tablet_service.cc:1505] Processing CreateTablet for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 (DEFAULT_TABLE table=default.Table_403e5819717949e8bffa1ed844a68964 [id=faaedb3e871c491691ecec42c3ab9d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.154390 1202 ts_tablet_manager.cc:1434] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:30.154676 1202 tablet_bootstrap.cc:492] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:30.155701 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.156194 1202 tablet_bootstrap.cc:654] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.156692 2735 consensus_queue.cc:1048] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:30.054742 1206 consensus_queue.cc:260] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:30.160538 851 consensus_peers.cc:597] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b70579ddf96f4499b4e4f235e8f69d67. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:30.161115 1018 tablet_service.cc:1505] Processing CreateTablet for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 (DEFAULT_TABLE table=default.Table_403e5819717949e8bffa1ed844a68964 [id=faaedb3e871c491691ecec42c3ab9d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.161504 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.162827 881 tablet_service.cc:1505] Processing CreateTablet for tablet c7885f21a369461c8c80ff18d369e2c6 (DEFAULT_TABLE table=default.Table_7fbc660486544d9fb711dcdca0660237 [id=a20045f442604c45a67b9b06f1eb4610]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.163328 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7885f21a369461c8c80ff18d369e2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.167423 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_529fb2317e754462b7ae49d9304e5402"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.167555 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_42e9fda7b7d14640a5804197bfb41683"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.168195 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_529fb2317e754462b7ae49d9304e5402 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:30.168309 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42e9fda7b7d14640a5804197bfb41683 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.169967 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e" } new_table_name: "default.Table_9060872cac6842548b9f1f80d16dbe19" modify_external_catalogs: true
I20251214 20:52:30.170331 883 tablet_service.cc:1505] Processing CreateTablet for tablet 4106fb80b4304735bc3faba7a7bf342f (DEFAULT_TABLE table=default.Table_b567effdb6834f7b87ce4aba9e555913 [id=c216b38d4fce484da23d096493ac2934]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.170710 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4106fb80b4304735bc3faba7a7bf342f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.171347 452 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_a2d3b2edaadd495c8df2d78ef9b8ed8e [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 1 got 0
I20251214 20:52:30.178339 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 18e95ff0e0da4c7680f594ba31cdc3a1
I20251214 20:52:30.181618 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_176fae22b3bf4a159ac799ec6339ea16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.182446 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_176fae22b3bf4a159ac799ec6339ea16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.188405 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet c249c096a41f4b4093eaf9afc6361249
I20251214 20:52:30.189292 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_2d61f26706854e2cbf1555c7ecd45b2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.189841 1203 tablet_bootstrap.cc:654] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:30.189950 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d61f26706854e2cbf1555c7ecd45b2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:30.190233 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 119446us
I20251214 20:52:30.190735 2652 raft_consensus.cc:2749] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.195556 450 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet a6df388bdfbc40c183c4b14af2336a6a
I20251214 20:52:30.196081 450 catalog_manager.cc:6724] ReplaceTablet: tablet a6df388bdfbc40c183c4b14af2336a6a deleted and replaced by tablet 74d2f8e5481d40b39cc4df34b0d4f819
I20251214 20:52:30.196497 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_81b328890d6748ea8a95ac3cd696c9dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.197108 1982 ts_tablet_manager.cc:1950] T 2accf8a160584c82bb695a52e3f2b05c P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
W20251214 20:52:30.197106 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81b328890d6748ea8a95ac3cd696c9dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.197386 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 404) took 2299 ms. Trace:
I20251214 20:52:30.197453 1982 rpcz_store.cc:276] 1214 20:52:27.897463 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.897606 (+ 143us) service_pool.cc:225] Handling call
1214 20:52:29.804367 (+1906761us) ts_tablet_manager.cc:1151] Deleting tablet 2accf8a160584c82bb695a52e3f2b05c
1214 20:52:29.804376 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:29.993706 (+189330us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.141638 (+147932us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.197371 (+ 55733us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1874387}
I20251214 20:52:30.197783 1982 tablet_replica.cc:333] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:30.197953 1982 raft_consensus.cc:2243] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.198086 1982 raft_consensus.cc:2272] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.198065 454 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_7e2ed71ac05340a1be0ebd6af1b9dcc4" } new_table_name: "default.Table_d61784eed0b84f8caa485dbe70cdc366" modify_external_catalogs: true
I20251214 20:52:30.198441 446 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 96dc4bdabb234c3fa4c26bd1633e783d
I20251214 20:52:30.198571 1982 ts_tablet_manager.cc:1916] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.199240 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_8d0c148da21f4137b09c9631e0b3f9bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.199724 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d0c148da21f4137b09c9631e0b3f9bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.200508 1206 ts_tablet_manager.cc:1434] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.147s user 0.001s sys 0.001s
I20251214 20:52:30.201308 883 tablet_service.cc:1505] Processing CreateTablet for tablet 6bf7317683ff4a52b3503687f4131ad3 (DEFAULT_TABLE table=default.Table_6588026e2d974db9b72d2b81ce8310cb [id=483610ffc7db4e848d0756374015974f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.202105 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet a6df388bdfbc40c183c4b14af2336a6a with delete_type TABLET_DATA_DELETED (replaced by tablet 74d2f8e5481d40b39cc4df34b0d4f819) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.202252 876 tablet_service.cc:1505] Processing CreateTablet for tablet 83da479367fe4b949f55bc6756883892 (DEFAULT_TABLE table=default.Table_f7d781182d07441a824d887e07286dc3 [id=6170186dc9cc4f4e93394530eb7c3a56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.202958 892 tablet_service.cc:1505] Processing CreateTablet for tablet e1dbaa39efa04086a22cc3908ed691d4 (DEFAULT_TABLE table=default.Table_8f475bc0f2dd4602b5f8e057241c3c16 [id=4390cb960d564bb5bc7e0cefa5bf3004]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.203388 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1dbaa39efa04086a22cc3908ed691d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.203917 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83da479367fe4b949f55bc6756883892. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.204001 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 44c61824bacd406280ba03357f9f5b1d
I20251214 20:52:30.204732 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_1225d07e2e7a4aa1b60557b64de1f35e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.205355 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1225d07e2e7a4aa1b60557b64de1f35e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.206153 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 96dc4bdabb234c3fa4c26bd1633e783d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.206908 746 tablet_service.cc:1505] Processing CreateTablet for tablet 2ea3c1e1ca5a41df8a01546bc08c3e23 (DEFAULT_TABLE table=default.Table_80391b5a5aa9425794dfb7997c659009 [id=893edd42fa04409b9d8e4a4e46d38ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.207307 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea3c1e1ca5a41df8a01546bc08c3e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.208815 1203 tablet_bootstrap.cc:492] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:30.208914 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c61824bacd406280ba03357f9f5b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.208992 1203 ts_tablet_manager.cc:1403] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20251214 20:52:30.209740 1203 raft_consensus.cc:359] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.209923 1203 raft_consensus.cc:385] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.209980 1203 raft_consensus.cc:740] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.203914 1977 ts_tablet_manager.cc:1929] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.213095 1977 log.cc:1199] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/37a33cc9bb784f72956ac9f5493a0c3d
I20251214 20:52:30.213451 1977 ts_tablet_manager.cc:1950] T 37a33cc9bb784f72956ac9f5493a0c3d P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:30.213800 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 414) took 2314 ms. Trace:
I20251214 20:52:30.213903 1977 rpcz_store.cc:276] 1214 20:52:27.899775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.900028 (+ 253us) service_pool.cc:225] Handling call
1214 20:52:30.036752 (+2136724us) ts_tablet_manager.cc:1151] Deleting tablet 37a33cc9bb784f72956ac9f5493a0c3d
1214 20:52:30.036763 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.122252 (+ 85489us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.203377 (+ 81125us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.213777 (+ 10400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2135940}
I20251214 20:52:30.214203 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet c249c096a41f4b4093eaf9afc6361249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.215591 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bf7317683ff4a52b3503687f4131ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.219821 448 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_79c0f2d73838484eba9470ccec3e90b9" } new_table_name: "default.Table_b289a25bf8174d9199c90212ce70271d" modify_external_catalogs: true
I20251214 20:52:30.220088 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 18e95ff0e0da4c7680f594ba31cdc3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.223723 1202 tablet_bootstrap.cc:492] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:30.223882 1202 ts_tablet_manager.cc:1403] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20251214 20:52:30.224692 2731 tablet.cc:1722] T d9f198ef4dfa4f5791144b9a24fe693f P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:30.225574 1202 raft_consensus.cc:359] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.225769 1202 raft_consensus.cc:385] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.225831 1202 raft_consensus.cc:740] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.225833 1856 ts_tablet_manager.cc:1929] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.226063 1202 consensus_queue.cc:260] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.228240 892 tablet_service.cc:1505] Processing CreateTablet for tablet 780be07e6f5d425d9605402b6db63012 (DEFAULT_TABLE table=default.Table_ee7a207364d74e6b81d6a72e57cacbd5 [id=b251a30885f6435fbfa430b6b50aee7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.229373 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9ba7f3a5c10e401b9edd530f37db3e17" } modify_external_catalogs: true
I20251214 20:52:30.229442 2730 tablet.cc:1722] T 738668adf2874d6894db6fa875f6990c P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:30.229584 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9ba7f3a5c10e401b9edd530f37db3e17" } modify_external_catalogs: true
I20251214 20:52:30.225934 1856 log.cc:1199] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/1dca369dc5f54a4ea1783e334c5670d6
I20251214 20:52:30.229979 1856 ts_tablet_manager.cc:1950] T 1dca369dc5f54a4ea1783e334c5670d6 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:30.235255 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 416) took 2335 ms. Trace:
I20251214 20:52:30.236455 1203 consensus_queue.cc:260] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.236641 752 tablet_service.cc:1505] Processing CreateTablet for tablet 5f251ff9b3fa4fe8ababab456733bcdd (DEFAULT_TABLE table=default.Table_b9ef85d4768d42d28d635a7c3eac5f4c [id=7be3dc7e56824beb9f13c242645bdff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.237748 1203 ts_tablet_manager.cc:1434] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20251214 20:52:30.242105 739 tablet_service.cc:1505] Processing CreateTablet for tablet 6404321f391944769986905a12fbab1c (DEFAULT_TABLE table=default.Table_4abdf07a08814d65acc20b6c32bfd415 [id=aeb3160d155949699e454f389fd07aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.246752 747 tablet_service.cc:1505] Processing CreateTablet for tablet f6d51d07f7e9450a844be9afddb60d03 (DEFAULT_TABLE table=default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.247264 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d51d07f7e9450a844be9afddb60d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.250635 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f251ff9b3fa4fe8ababab456733bcdd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:30.256155 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 719d157ac8f74eb7911df05ab5e4a3d5 consensus state has no leader
I20251214 20:52:30.257543 883 tablet_service.cc:1505] Processing CreateTablet for tablet de9c830cf9c74cff88cb6f42cbb5f7cd (DEFAULT_TABLE table=default.Table_f9499a2114f44353aaf65b35ed56f119 [id=7dd81c38dde94205a84d6f7841d2bb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.257969 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de9c830cf9c74cff88cb6f42cbb5f7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.260630 881 tablet_service.cc:1505] Processing CreateTablet for tablet d48c6789e5724e989900d3351ecde3ab (DEFAULT_TABLE table=default.Table_b6011f3e00584819944d165d653d06ac [id=103545908ce040858f00aed970b27479]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.260550 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_1aadd2d8580e419fb5a3767b09353c25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.261008 453 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_00bef63ea36c4a7585c2832be1454b68" } new_table_name: "default.Table_53bf327c948f4cebbe6096266ac301f8" modify_external_catalogs: true
W20251214 20:52:30.261193 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1aadd2d8580e419fb5a3767b09353c25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.263085 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_09270706a48c42fea27376b3ef3a0746"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.263643 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09270706a48c42fea27376b3ef3a0746 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.271981 740 tablet_service.cc:1505] Processing CreateTablet for tablet 3373921f73b64fae954ea7c5d5f1d93f (DEFAULT_TABLE table=default.Table_5cda3bedf2d24fc2b279c049146aa2c5 [id=10ef2e196edd432aa21fd7d31fa4366f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.272420 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3373921f73b64fae954ea7c5d5f1d93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.275663 451 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e90d11d69b8d4fd49150aa67a8190e5b (table default.Table_c5bdb81886bf4361af53fd1a3b3b0b16 [id=b3df2580f4a74189aa06f021d6b4ce71]). Expected version 1 got 0
I20251214 20:52:30.275911 451 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 9da5e98c6f4843449886176e9fcfa8e1 (table default.Table_55f303d33ec84a269232fc1ba8f2d182 [id=5ded9fe825ae4308b5bd4b21c16634c8]). Expected version 1 got 0
I20251214 20:52:30.276166 451 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_9060872cac6842548b9f1f80d16dbe19 [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 2 got 0
I20251214 20:52:30.276408 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 18e95ff0e0da4c7680f594ba31cdc3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:30.276573 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 17550eb0ee6b476f8a8c304451f678b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:30.276679 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 96dc4bdabb234c3fa4c26bd1633e783d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:30.276844 738 tablet_service.cc:1552] Processing DeleteTablet for tablet c249c096a41f4b4093eaf9afc6361249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:30.278908 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_6eb267b64a6a485bb607876002c23ec4" } new_table_name: "default.Table_90e94bab31e54a9cb8b567622aa4e37d" modify_external_catalogs: true
I20251214 20:52:30.246428 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 477) took 1820 ms. Trace:
I20251214 20:52:30.249181 1103 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34819
I20251214 20:52:30.279983 1856 rpcz_store.cc:276] 1214 20:52:28.425760 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.425839 (+ 79us) service_pool.cc:225] Handling call
1214 20:52:29.925126 (+1499287us) ts_tablet_manager.cc:1151] Deleting tablet 1dca369dc5f54a4ea1783e334c5670d6
1214 20:52:29.925137 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.211696 (+286559us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.225695 (+ 13999us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.246394 (+ 20699us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1499082}
I20251214 20:52:30.280287 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a000951f61948ac85fdd431f3d162c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.280630 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 1dca369dc5f54a4ea1783e334c5670d6 (table default.Table_4ad0cc66006940d2abacd388e227075a [id=3f84c0b6401d4ff18c79c0d38e26ca97]) successfully deleted
I20251214 20:52:30.280973 1856 tablet_replica.cc:333] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:30.281167 1856 raft_consensus.cc:2243] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:30.281497 1856 raft_consensus.cc:2272] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.282485 1856 ts_tablet_manager.cc:1916] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.289793 2397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41536 (request call id 0) took 4044 ms. Trace:
I20251214 20:52:30.290808 1856 ts_tablet_manager.cc:1929] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:30.290706 2645 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27224391721467a8f46e467a980b37a. Latest consensus state: current_term: 4 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:30.291004 1856 log.cc:1199] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/47e813373fbd482298312ead4f037c01
I20251214 20:52:30.291028 2645 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:30.291235 2689 raft_consensus.cc:493] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c27224391721467a8f46e467a980b37a)
I20251214 20:52:30.291394 1856 ts_tablet_manager.cc:1950] T 47e813373fbd482298312ead4f037c01 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:30.291478 2689 raft_consensus.cc:515] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
W20251214 20:52:30.295372 712 consensus_peers.cc:597] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf8643c7e4094f529cd55656c7b3edb6. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:30.295574 712 leader_election.cc:304] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:30.296025 2714 raft_consensus.cc:2749] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.297161 1982 ts_tablet_manager.cc:1929] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.297261 1982 log.cc:1199] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/b315744c8cd043c5a8bdceec73a7e28b
I20251214 20:52:30.297574 1982 ts_tablet_manager.cc:1950] T b315744c8cd043c5a8bdceec73a7e28b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:30.297863 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 403) took 2400 ms. Trace:
I20251214 20:52:30.297933 1982 rpcz_store.cc:276] 1214 20:52:27.897339 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.897516 (+ 177us) service_pool.cc:225] Handling call
1214 20:52:30.197731 (+2300215us) ts_tablet_manager.cc:1151] Deleting tablet b315744c8cd043c5a8bdceec73a7e28b
1214 20:52:30.197740 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.239008 (+ 41268us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.297045 (+ 58037us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.297848 (+ 803us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2262492}
I20251214 20:52:30.298086 1203 tablet_bootstrap.cc:492] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:30.298187 2689 leader_election.cc:290] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:30.298318 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 408) took 2353 ms. Trace:
I20251214 20:52:30.298378 1982 rpcz_store.cc:276] 1214 20:52:27.945005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.945060 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:30.298189 (+2353129us) ts_tablet_manager.cc:1151] Deleting tablet c3453fc996fb4d2b8cf0f80152098c43
1214 20:52:30.298197 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.298304 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352948}
I20251214 20:52:30.298601 1982 tablet_replica.cc:333] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:30.298769 1982 raft_consensus.cc:2243] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.298916 1982 raft_consensus.cc:2272] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.301424 1203 tablet_bootstrap.cc:654] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.301891 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 2accf8a160584c82bb695a52e3f2b05c (table default.Table_b4334a1d99a54c2eabe2fb909563313c [id=d31b208e2e284575ad429878129a28f5]) successfully deleted
I20251214 20:52:30.301970 2735 raft_consensus.cc:493] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.302098 2735 raft_consensus.cc:515] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.302338 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet b315744c8cd043c5a8bdceec73a7e28b (table default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]) successfully deleted
W20251214 20:52:30.302603 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c3453fc996fb4d2b8cf0f80152098c43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3453fc996fb4d2b8cf0f80152098c43 already in progress: creating tablet
I20251214 20:52:30.302778 2735 leader_election.cc:290] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:30.302978 776 raft_consensus.cc:3060] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:30.303169 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6404321f391944769986905a12fbab1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.303742 775 raft_consensus.cc:3060] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:30.236135 1977 rpcz_store.cc:276] 1214 20:52:27.899975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.900718 (+ 743us) service_pool.cc:225] Handling call
1214 20:52:30.214207 (+2313489us) ts_tablet_manager.cc:1151] Deleting tablet 2d164747cb294bd4b126b4420231867d
1214 20:52:30.214217 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.235221 (+ 21004us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2313347}
I20251214 20:52:30.304634 876 tablet_service.cc:1505] Processing CreateTablet for tablet 556ac6b97fab4ea5a43089c0f96c38e5 (DEFAULT_TABLE table=default.Table_8f5d31355346492989caa148a8f1440b [id=ec2fd20884ff4bdca7be9d32d3bfe9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.305052 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 780be07e6f5d425d9605402b6db63012. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.305178 1202 ts_tablet_manager.cc:1434] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.081s user 0.003s sys 0.000s
I20251214 20:52:30.306289 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48c6789e5724e989900d3351ecde3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.307562 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 478) took 1820 ms. Trace:
I20251214 20:52:30.307723 1856 rpcz_store.cc:276] 1214 20:52:28.486942 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.487006 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:30.280313 (+1793307us) ts_tablet_manager.cc:1151] Deleting tablet 47e813373fbd482298312ead4f037c01
1214 20:52:30.280925 (+ 612us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.285657 (+ 4732us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.290688 (+ 5031us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.307538 (+ 16850us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1793055}
I20251214 20:52:30.308130 1856 tablet_replica.cc:333] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:30.308280 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 47e813373fbd482298312ead4f037c01 (table default.Table_a60363e46077464b968f9187a8c7a8fc [id=708908b1be53473f8c5f7e191128e7e2]) successfully deleted
I20251214 20:52:30.308365 1856 raft_consensus.cc:2243] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.308537 1856 raft_consensus.cc:2272] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.308637 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c61824bacd406280ba03357f9f5b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:30.309368 1856 ts_tablet_manager.cc:1916] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.310863 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f84e9dfc23d34eb0a379e8ac5a442e59" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:30.311424 991 leader_election.cc:343] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.311494 1202 tablet_bootstrap.cc:492] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:30.311825 1977 tablet_replica.cc:333] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:30.311915 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 556ac6b97fab4ea5a43089c0f96c38e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.312018 1977 raft_consensus.cc:2243] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.313524 891 tablet_service.cc:1505] Processing CreateTablet for tablet 19008711a4754ca7b38a3ef2f0cdbf1d (DEFAULT_TABLE table=default.Table_a07fe706644d4c63b966f0751b757642 [id=8845cef019b44676a83a3173c5df0bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.313872 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19008711a4754ca7b38a3ef2f0cdbf1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.315635 880 tablet_service.cc:1505] Processing CreateTablet for tablet 1ee28c4ac91448629c0bddb3143f7df0 (DEFAULT_TABLE table=default.Table_bf1d2eb618b64a4e86cd121824fec0cb [id=1c14038af47c43c28ec8370fba0a14df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.315960 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ee28c4ac91448629c0bddb3143f7df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.318531 879 tablet_service.cc:1505] Processing CreateTablet for tablet ae54e7a5104e4cd588a254551fc52477 (DEFAULT_TABLE table=default.Table_68c79df4e9e0446f81c8500bcfa423a6 [id=7e9bbbe22677488d9509dc201b2d7234]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.318859 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae54e7a5104e4cd588a254551fc52477. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.318907 1856 ts_tablet_manager.cc:1929] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.319036 1856 log.cc:1199] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/8d9f74fb2c6244a6887e3c922262bbab
I20251214 20:52:30.319296 1977 raft_consensus.cc:2272] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.319367 1856 ts_tablet_manager.cc:1950] T 8d9f74fb2c6244a6887e3c922262bbab P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:30.319962 1977 ts_tablet_manager.cc:1916] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.321163 1202 tablet_bootstrap.cc:654] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.325659 884 tablet_service.cc:1505] Processing CreateTablet for tablet 69f9d5838e544002ab49f0ce21b6f7ef (DEFAULT_TABLE table=default.Table_9ff099dcfac14636a9dcaa70207c86a6 [id=4a1816b57ff84e459394e59b683d8ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.326069 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f9d5838e544002ab49f0ce21b6f7ef. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:30.326771 442 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6302145a6e7b4297a58d9d7f3ad02dce (table default.Table_1be39fb576af4b2b9e2606b495619863 [id=54e0182cd3b9461aa593a11aa7709002], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6302145a6e7b4297a58d9d7f3ad02dce consensus state has no leader
I20251214 20:52:30.327692 886 tablet_service.cc:1505] Processing CreateTablet for tablet 5afb7fb4f0a44b0092fbef5bbec9f396 (DEFAULT_TABLE table=default.Table_87f9c82a6a89490391b5c78001639d2b [id=8203de75aba94c3f8561f18adaaa825e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.328022 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afb7fb4f0a44b0092fbef5bbec9f396. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.328903 878 tablet_service.cc:1505] Processing CreateTablet for tablet fd3a49d6acae40d099cdd99981ce9d46 (DEFAULT_TABLE table=default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.329479 885 tablet_service.cc:1505] Processing CreateTablet for tablet 700ed62ef2e84719a7a8e9879feaf45c (DEFAULT_TABLE table=default.Table_65d3039202a645e899e5f752e1e3db42 [id=b6bebe3ab7e2449dafc61fdda3e3c38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.329826 887 tablet_service.cc:1505] Processing CreateTablet for tablet c5478cba0bf24f8c969fd1b0998b9db3 (DEFAULT_TABLE table=default.Table_0f409000b9514206a08757040e81a6a9 [id=5692c4a553044543ab275cff0f7a0566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.329982 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 700ed62ef2e84719a7a8e9879feaf45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.330631 877 tablet_service.cc:1505] Processing CreateTablet for tablet 2ea3c1e1ca5a41df8a01546bc08c3e23 (DEFAULT_TABLE table=default.Table_80391b5a5aa9425794dfb7997c659009 [id=893edd42fa04409b9d8e4a4e46d38ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.330940 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ea3c1e1ca5a41df8a01546bc08c3e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.304435 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f84e9dfc23d34eb0a379e8ac5a442e59" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:30.332638 753 tablet_service.cc:1552] Processing DeleteTablet for tablet a6df388bdfbc40c183c4b14af2336a6a with delete_type TABLET_DATA_DELETED (replaced by tablet 74d2f8e5481d40b39cc4df34b0d4f819) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:30.336083 989 leader_election.cc:343] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.336212 989 leader_election.cc:304] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:30.336809 2735 raft_consensus.cc:2749] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.338502 894 tablet_service.cc:1505] Processing CreateTablet for tablet 3373921f73b64fae954ea7c5d5f1d93f (DEFAULT_TABLE table=default.Table_5cda3bedf2d24fc2b279c049146aa2c5 [id=10ef2e196edd432aa21fd7d31fa4366f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.339816 2712 raft_consensus.cc:493] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.339938 2712 raft_consensus.cc:515] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.340037 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 37a33cc9bb784f72956ac9f5493a0c3d (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]) successfully deleted
W20251214 20:52:30.340471 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 2d164747cb294bd4b126b4420231867d because the tablet was not found. No further retry: Not found: Tablet not found: 2d164747cb294bd4b126b4420231867d
I20251214 20:52:30.340524 2712 leader_election.cc:290] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:30.341605 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "453cf734f8b34b59aa246d618ffeade5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:30.342557 848 leader_election.cc:343] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.343066 1203 tablet_bootstrap.cc:492] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
W20251214 20:52:30.343353 851 leader_election.cc:343] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 453cf734f8b34b59aa246d618ffeade5
I20251214 20:52:30.343467 851 leader_election.cc:304] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:30.343865 2712 raft_consensus.cc:2749] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.344416 2738 tablet.cc:1722] T d9f198ef4dfa4f5791144b9a24fe693f P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:30.346254 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd3a49d6acae40d099cdd99981ce9d46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.343694 2739 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:30.349089 2739 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:30.350147 1206 tablet_bootstrap.cc:492] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:30.356567 895 tablet_service.cc:1505] Processing CreateTablet for tablet f6d51d07f7e9450a844be9afddb60d03 (DEFAULT_TABLE table=default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.357040 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d51d07f7e9450a844be9afddb60d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.359422 893 tablet_service.cc:1505] Processing CreateTablet for tablet d39e245bc5a04c1186d0973645aa1b54 (DEFAULT_TABLE table=default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.359748 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d39e245bc5a04c1186d0973645aa1b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.363165 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5478cba0bf24f8c969fd1b0998b9db3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.381994 1203 ts_tablet_manager.cc:1403] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.084s user 0.007s sys 0.003s
I20251214 20:52:30.384682 1206 tablet_bootstrap.cc:654] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.387301 1203 raft_consensus.cc:359] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.393615 1982 ts_tablet_manager.cc:1916] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.405624 1203 raft_consensus.cc:385] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.405942 1203 raft_consensus.cc:740] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.406349 1203 consensus_queue.cc:260] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.407675 1203 ts_tablet_manager.cc:1434] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251214 20:52:30.409545 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 479) took 1921 ms. Trace:
I20251214 20:52:30.409668 1856 rpcz_store.cc:276] 1214 20:52:28.487700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.487752 (+ 52us) service_pool.cc:225] Handling call
1214 20:52:30.308071 (+1820319us) ts_tablet_manager.cc:1151] Deleting tablet 8d9f74fb2c6244a6887e3c922262bbab
1214 20:52:30.308082 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.315610 (+ 7528us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.318760 (+ 3150us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.409522 (+ 90762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1820160}
I20251214 20:52:30.410161 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 487) took 1533 ms. Trace:
I20251214 20:52:30.410233 1856 rpcz_store.cc:276] 1214 20:52:28.876413 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.876475 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:30.410011 (+1533536us) ts_tablet_manager.cc:1151] Deleting tablet 1e185d107619414a8bcc15ad496547dc
1214 20:52:30.410021 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.410145 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1507576}
I20251214 20:52:30.410558 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 491) took 1466 ms. Trace:
I20251214 20:52:30.410625 1856 rpcz_store.cc:276] 1214 20:52:28.944606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.944673 (+ 67us) service_pool.cc:225] Handling call
1214 20:52:30.410463 (+1465790us) ts_tablet_manager.cc:1151] Deleting tablet 3915abc53b084854912a8e4119eefc7e
1214 20:52:30.410472 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.410544 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1465572}
I20251214 20:52:30.410723 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 8d9f74fb2c6244a6887e3c922262bbab (table default.Table_025651c33fe34ab1a55af7368f32fa04 [id=e52f1b7d812d41cb80078ce966183aa8]) successfully deleted
I20251214 20:52:30.411054 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 493) took 1427 ms. Trace:
W20251214 20:52:30.411075 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 1e185d107619414a8bcc15ad496547dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e185d107619414a8bcc15ad496547dc already in progress: creating tablet
I20251214 20:52:30.411150 1856 rpcz_store.cc:276] 1214 20:52:28.984068 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.984445 (+ 377us) service_pool.cc:225] Handling call
1214 20:52:30.410951 (+1426506us) ts_tablet_manager.cc:1151] Deleting tablet 6b389a63a58d438db41a343db6e04a88
1214 20:52:30.410960 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.411040 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1413681}
I20251214 20:52:30.411379 1856 tablet_replica.cc:333] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:30.411597 1856 raft_consensus.cc:2243] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.411814 1856 raft_consensus.cc:2272] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.412462 1856 ts_tablet_manager.cc:1916] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:30.415295 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 3915abc53b084854912a8e4119eefc7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3915abc53b084854912a8e4119eefc7e already in progress: creating tablet
W20251214 20:52:30.415604 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 6b389a63a58d438db41a343db6e04a88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b389a63a58d438db41a343db6e04a88 already in progress: creating tablet
I20251214 20:52:30.419921 1203 tablet_bootstrap.cc:492] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:30.421676 1203 tablet_bootstrap.cc:654] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:30.421772 454 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59ccb6c3b35044dc913cce74b5802e38 consensus state has no leader
I20251214 20:52:30.423410 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet cbefd8f111084785bafd7e74ac443fd0
I20251214 20:52:30.424176 446 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_edb8b6390dba4303b887b41508f5f525" } modify_external_catalogs: true
I20251214 20:52:30.424372 446 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_edb8b6390dba4303b887b41508f5f525" } modify_external_catalogs: true
I20251214 20:52:30.425916 455 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_de80fc127aa549ca931866a310858ddb" } modify_external_catalogs: true
I20251214 20:52:30.426301 455 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_de80fc127aa549ca931866a310858ddb" } modify_external_catalogs: true
I20251214 20:52:30.426587 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_2fca6840abf741ed94baef488b3640dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.426798 851 consensus_peers.cc:597] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d9f74fb2c6244a6887e3c922262bbab. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:30.427234 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fca6840abf741ed94baef488b3640dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.427870 1856 ts_tablet_manager.cc:1929] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.428025 1856 log.cc:1199] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/25dcf491653147e7831217c1ec7a2e0c
I20251214 20:52:30.428333 1856 ts_tablet_manager.cc:1950] T 25dcf491653147e7831217c1ec7a2e0c P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:30.428683 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 494) took 1422 ms. Trace:
I20251214 20:52:30.428784 1856 rpcz_store.cc:276] 1214 20:52:29.005793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.006090 (+ 297us) service_pool.cc:225] Handling call
1214 20:52:30.411333 (+1405243us) ts_tablet_manager.cc:1151] Deleting tablet 25dcf491653147e7831217c1ec7a2e0c
1214 20:52:30.411340 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.417067 (+ 5727us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.427727 (+ 10660us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.428665 (+ 938us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1397358}
I20251214 20:52:30.429566 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 495) took 1402 ms. Trace:
I20251214 20:52:30.430109 1856 rpcz_store.cc:276] 1214 20:52:29.026744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.026998 (+ 254us) service_pool.cc:225] Handling call
1214 20:52:30.429425 (+1402427us) ts_tablet_manager.cc:1151] Deleting tablet a88207c8cc5f4a6886de36c82c6b4c2f
1214 20:52:30.429434 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.429549 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1397338}
I20251214 20:52:30.431047 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 497) took 1385 ms. Trace:
I20251214 20:52:30.431440 739 tablet_service.cc:1552] Processing DeleteTablet for tablet cbefd8f111084785bafd7e74ac443fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:30.431764 1856 rpcz_store.cc:276] 1214 20:52:29.045964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.046024 (+ 60us) service_pool.cc:225] Handling call
1214 20:52:30.430942 (+1384918us) ts_tablet_manager.cc:1151] Deleting tablet b2bea617954c475bac7e9b8719faa028
1214 20:52:30.430952 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.431031 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1384676}
I20251214 20:52:30.432030 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 25dcf491653147e7831217c1ec7a2e0c (table default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7]) successfully deleted
W20251214 20:52:30.432381 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a88207c8cc5f4a6886de36c82c6b4c2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a88207c8cc5f4a6886de36c82c6b4c2f already in progress: creating tablet
I20251214 20:52:30.432824 1856 tablet_replica.cc:333] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:30.432978 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b2bea617954c475bac7e9b8719faa028 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2bea617954c475bac7e9b8719faa028 already in progress: creating tablet
I20251214 20:52:30.433233 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet cbefd8f111084785bafd7e74ac443fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.433626 1856 raft_consensus.cc:2243] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:30.434218 1856 raft_consensus.cc:2272] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.436939 1856 ts_tablet_manager.cc:1916] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.439409 1203 tablet_bootstrap.cc:492] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:30.441058 1203 ts_tablet_manager.cc:1403] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20251214 20:52:30.442056 1203 raft_consensus.cc:359] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.443195 1203 raft_consensus.cc:385] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.443408 1203 raft_consensus.cc:740] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.452924 1203 consensus_queue.cc:260] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.458447 1203 ts_tablet_manager.cc:1434] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
W20251214 20:52:30.461560 851 consensus_peers.cc:597] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 956e4b4493f24109ab1251dde0a4cfc0. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:30.463692 889 tablet_service.cc:1552] Processing DeleteTablet for tablet bf37c50e511f454ca4887c7f6ca1a746 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:30.464103 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3373921f73b64fae954ea7c5d5f1d93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.464207 882 tablet_service.cc:1505] Processing CreateTablet for tablet 86ede5e41a03485082f4531fc42dfe56 (DEFAULT_TABLE table=default.Table_ee3c57e398fd468fa1e4e7cc22f6747a [id=ce8357dfd1024446a69a3ee196ab4b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.464532 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ede5e41a03485082f4531fc42dfe56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.465962 1203 tablet_bootstrap.cc:492] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:30.466121 890 tablet_service.cc:1505] Processing CreateTablet for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 (DEFAULT_TABLE table=default.Table_403e5819717949e8bffa1ed844a68964 [id=faaedb3e871c491691ecec42c3ab9d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.466459 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.466948 889 tablet_service.cc:1505] Processing CreateTablet for tablet 6404321f391944769986905a12fbab1c (DEFAULT_TABLE table=default.Table_4abdf07a08814d65acc20b6c32bfd415 [id=aeb3160d155949699e454f389fd07aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.467515 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6404321f391944769986905a12fbab1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.469216 888 tablet_service.cc:1505] Processing CreateTablet for tablet 5f251ff9b3fa4fe8ababab456733bcdd (DEFAULT_TABLE table=default.Table_b9ef85d4768d42d28d635a7c3eac5f4c [id=7be3dc7e56824beb9f13c242645bdff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.469525 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f251ff9b3fa4fe8ababab456733bcdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.473783 1856 ts_tablet_manager.cc:1929] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:30.475759 1856 log.cc:1199] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/cbbfcef827ee4d548939aaf4783c8109
I20251214 20:52:30.476363 1856 ts_tablet_manager.cc:1950] T cbbfcef827ee4d548939aaf4783c8109 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:30.477658 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 498) took 1415 ms. Trace:
W20251214 20:52:30.478432 448 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a818ca230564b93ac165d43ff22deb1 consensus state has no leader
I20251214 20:52:30.478680 1856 rpcz_store.cc:276] 1214 20:52:29.062301 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.062368 (+ 67us) service_pool.cc:225] Handling call
1214 20:52:30.432428 (+1370060us) ts_tablet_manager.cc:1151] Deleting tablet cbbfcef827ee4d548939aaf4783c8109
1214 20:52:30.432437 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.445444 (+ 13007us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.473655 (+ 28211us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.477623 (+ 3968us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369806}
I20251214 20:52:30.479027 1856 tablet_replica.cc:333] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:30.479858 1856 raft_consensus.cc:2243] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.480134 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet cbbfcef827ee4d548939aaf4783c8109 (table default.Table_95769572161249d98020c723ac65b78e [id=906bc0748b9e44198eab3d4be4aced05]) successfully deleted
I20251214 20:52:30.480331 1856 raft_consensus.cc:2272] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.480626 2328 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" instance_seqno: 1765745526787922) as {username='slave'} at 127.0.104.1:55059; Asking this server to re-register.
I20251214 20:52:30.481144 1856 ts_tablet_manager.cc:1916] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.490996 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 3d949abef1cb41a9905482d81a98d2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:30.490949 1334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 722 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" is_pre_election: true
I20251214 20:52:30.493680 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 18e95ff0e0da4c7680f594ba31cdc3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:30.494026 894 tablet_service.cc:1552] Processing DeleteTablet for tablet c249c096a41f4b4093eaf9afc6361249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:30.494228 894 tablet_service.cc:1552] Processing DeleteTablet for tablet a6df388bdfbc40c183c4b14af2336a6a with delete_type TABLET_DATA_DELETED (replaced by tablet 74d2f8e5481d40b39cc4df34b0d4f819) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:30.494577 1856 ts_tablet_manager.cc:1929] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.494720 1856 log.cc:1199] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/06381efe75284e649fd6d1a9903cfa9b
I20251214 20:52:30.495003 1856 ts_tablet_manager.cc:1950] T 06381efe75284e649fd6d1a9903cfa9b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:30.496378 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 06381efe75284e649fd6d1a9903cfa9b (table default.Table_7bc0f81d22a643589e8a54ceb2477807 [id=db33edd24660463ea648be7da745e29c]) successfully deleted
W20251214 20:52:30.497668 453 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dbfe8ee68bf4a2c88489b1f81b36991 consensus state has no leader
W20251214 20:52:30.498468 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96a2b444608e42d687eeba2dc0ebb757 consensus state has no leader
I20251214 20:52:30.500130 455 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet bd1161eebe884fe6b347bf517269c047
W20251214 20:52:30.500583 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 8e11723c042a4637839f1b13bed0935a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e11723c042a4637839f1b13bed0935a already in progress: creating tablet
I20251214 20:52:30.501559 1018 tablet_service.cc:1552] Processing DeleteTablet for tablet bd1161eebe884fe6b347bf517269c047 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:40270
W20251214 20:52:30.501998 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet d2a57409d4f54f17be792e6f3f17d7b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2a57409d4f54f17be792e6f3f17d7b7 already in progress: creating tablet
I20251214 20:52:30.502238 748 tablet_service.cc:1552] Processing DeleteTablet for tablet bd1161eebe884fe6b347bf517269c047 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:57406
W20251214 20:52:30.502832 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b3e0a3bab7a47cb8b26f2094f5a1680 already in progress: creating tablet
W20251214 20:52:30.503954 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f98be57ce1b14f20a3afcb79979b6c22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f98be57ce1b14f20a3afcb79979b6c22 already in progress: creating tablet
I20251214 20:52:30.504415 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c5bdb81886bf4361af53fd1a3b3b0b16" } modify_external_catalogs: true
I20251214 20:52:30.504689 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_c5bdb81886bf4361af53fd1a3b3b0b16" } modify_external_catalogs: true
I20251214 20:52:30.505023 1856 tablet_replica.cc:333] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:30.505448 1856 raft_consensus.cc:2243] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.505622 1856 raft_consensus.cc:2272] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.506238 1856 ts_tablet_manager.cc:1916] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.506678 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_17c9057bb25b4cce9b1072a93f6ea38a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.507053 2712 raft_consensus.cc:493] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.507230 2712 raft_consensus.cc:515] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
W20251214 20:52:30.507339 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17c9057bb25b4cce9b1072a93f6ea38a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.507822 2712 leader_election.cc:290] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:30.508734 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_e277c40da9e24b7fabb8edd25ec3413d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.509232 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e277c40da9e24b7fabb8edd25ec3413d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.510603 448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9af6d4105a984471b526cd9d4d0941f8" } modify_external_catalogs: true
W20251214 20:52:30.510715 851 leader_election.cc:343] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 06381efe75284e649fd6d1a9903cfa9b
I20251214 20:52:30.510756 1977 ts_tablet_manager.cc:1929] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.510780 448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9af6d4105a984471b526cd9d4d0941f8" } modify_external_catalogs: true
I20251214 20:52:30.510883 1977 log.cc:1199] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/4d6e7306254044449f1127fb963ca5f0
W20251214 20:52:30.504710 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet f7f06eded6c340c58101fcc644ebcf50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7f06eded6c340c58101fcc644ebcf50 already in progress: creating tablet
I20251214 20:52:30.511263 880 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c61824bacd406280ba03357f9f5b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:30.511494 1977 ts_tablet_manager.cc:1950] T 4d6e7306254044449f1127fb963ca5f0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:30.511843 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_da61d1abe3f24498977b6cc43d2ed4df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.512375 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da61d1abe3f24498977b6cc43d2ed4df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.513135 449 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_8f475bc0f2dd4602b5f8e057241c3c16" } new_table_name: "default.Table_8f897cb4591a4d5fadbf2b1b7ee7408d" modify_external_catalogs: true
I20251214 20:52:30.513448 880 tablet_service.cc:1552] Processing DeleteTablet for tablet cbefd8f111084785bafd7e74ac443fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:30.514355 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 413) took 2614 ms. Trace:
I20251214 20:52:30.514464 1977 rpcz_store.cc:276] 1214 20:52:27.899676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.900874 (+ 1198us) service_pool.cc:225] Handling call
1214 20:52:30.304829 (+2403955us) ts_tablet_manager.cc:1151] Deleting tablet 4d6e7306254044449f1127fb963ca5f0
1214 20:52:30.311775 (+ 6946us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.479749 (+167974us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.510605 (+ 30856us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.514325 (+ 3720us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5376,"tablet-delete.queue_time_us":2403801}
I20251214 20:52:30.514957 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 419) took 2538 ms. Trace:
I20251214 20:52:30.515033 1977 rpcz_store.cc:276] 1214 20:52:27.976155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.976242 (+ 87us) service_pool.cc:225] Handling call
1214 20:52:30.514776 (+2538534us) ts_tablet_manager.cc:1151] Deleting tablet c3453fc996fb4d2b8cf0f80152098c43
1214 20:52:30.514789 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.514940 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2494265}
I20251214 20:52:30.515538 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 424) took 2347 ms. Trace:
I20251214 20:52:30.515828 1977 rpcz_store.cc:276] 1214 20:52:28.167590 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.168421 (+ 831us) service_pool.cc:225] Handling call
1214 20:52:30.515312 (+2346891us) ts_tablet_manager.cc:1151] Deleting tablet a0976ffcc649461fac74e20b5e402275
1214 20:52:30.515320 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.515522 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346657}
I20251214 20:52:30.516360 1856 ts_tablet_manager.cc:1929] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:30.516696 1977 tablet_replica.cc:333] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:30.516940 1977 raft_consensus.cc:2243] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.517118 1977 raft_consensus.cc:2272] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.518793 463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 722 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a" is_pre_election: true
I20251214 20:52:30.521695 2330 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "63848a9fb78a4013a6f1657621272d75" instance_seqno: 1765745527881897) as {username='slave'} at 127.0.104.3:38489; Asking this server to re-register.
I20251214 20:52:30.516345 2328 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" instance_seqno: 1765745527287656) as {username='slave'} at 127.0.104.2:41037; Asking this server to re-register.
I20251214 20:52:30.522269 1977 ts_tablet_manager.cc:1916] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.520367 2288 leader_election.cc:304] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6713a20a82b434f8a8bc122113e26ad; no voters: 7c48b8eaf30c4e2caa94a51d087406d4, c27224391721467a8f46e467a980b37a
I20251214 20:52:30.523006 2739 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 20:52:30.524374 1856 log.cc:1199] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/18e119bc44874aecb4e83c8d6040c8cd
I20251214 20:52:30.524570 962 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:30.524681 1103 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:30.525619 1982 ts_tablet_manager.cc:1929] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.529589 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 4d6e7306254044449f1127fb963ca5f0 (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]) successfully deleted
I20251214 20:52:30.529976 1982 log.cc:1199] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/2d164747cb294bd4b126b4420231867d
W20251214 20:52:30.530138 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c3453fc996fb4d2b8cf0f80152098c43 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3453fc996fb4d2b8cf0f80152098c43 already in progress: creating tablet
I20251214 20:52:30.525060 1856 ts_tablet_manager.cc:1950] T 18e119bc44874aecb4e83c8d6040c8cd P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:30.535694 1982 ts_tablet_manager.cc:1950] T 2d164747cb294bd4b126b4420231867d P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:30.536752 2330 ts_manager.cc:194] Registered new tserver with Master: 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:30.537120 1856 tablet_replica.cc:333] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:30.537493 1856 raft_consensus.cc:2243] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.537859 1856 raft_consensus.cc:2272] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.538501 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 409) took 2550 ms. Trace:
W20251214 20:52:30.538939 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a0976ffcc649461fac74e20b5e402275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0976ffcc649461fac74e20b5e402275 already in progress: creating tablet
I20251214 20:52:30.538980 1856 ts_tablet_manager.cc:1916] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.541996 2703 raft_consensus.cc:493] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.542168 879 tablet_service.cc:1552] Processing DeleteTablet for tablet bd1161eebe884fe6b347bf517269c047 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:30.494421 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 96dc4bdabb234c3fa4c26bd1633e783d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:30.542883 2703 raft_consensus.cc:515] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.544472 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1835374cfa48939d72476870accc3b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:30.545061 991 leader_election.cc:343] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.545300 2703 leader_election.cc:290] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:30.546525 824 heartbeater.cc:461] Registering TS with master...
I20251214 20:52:30.547474 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "06381efe75284e649fd6d1a9903cfa9b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:30.548154 848 leader_election.cc:343] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.538901 1982 rpcz_store.cc:276] 1214 20:52:27.987611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.987686 (+ 75us) service_pool.cc:225] Handling call
1214 20:52:30.298561 (+2310875us) ts_tablet_manager.cc:1151] Deleting tablet 2d164747cb294bd4b126b4420231867d
1214 20:52:30.298569 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.396528 (+ 97959us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.525504 (+128976us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.538482 (+ 12978us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2309953}
I20251214 20:52:30.553510 1982 tablet_replica.cc:333] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:30.553776 1982 raft_consensus.cc:2243] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.553987 1203 tablet_bootstrap.cc:654] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.554008 1982 raft_consensus.cc:2272] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.554653 1982 ts_tablet_manager.cc:1916] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.554618 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 2d164747cb294bd4b126b4420231867d (table default.Table_c488616dc265498088254d401a3cc7db [id=57197e3100144ed8888c7c06dfcd00a1]) successfully deleted
I20251214 20:52:30.559868 1203 tablet_bootstrap.cc:492] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:30.560127 1203 ts_tablet_manager.cc:1403] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.001s
I20251214 20:52:30.561070 1203 raft_consensus.cc:359] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.561396 1203 raft_consensus.cc:385] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.561496 1203 raft_consensus.cc:740] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.561606 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1835374cfa48939d72476870accc3b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:30.561806 1203 consensus_queue.cc:260] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:30.562103 989 leader_election.cc:343] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.562234 989 leader_election.cc:304] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:30.562458 1203 ts_tablet_manager.cc:1434] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:30.562664 2703 raft_consensus.cc:2749] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.563333 2747 raft_consensus.cc:493] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.563557 2747 raft_consensus.cc:515] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.564718 2747 leader_election.cc:290] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:30.571725 1203 tablet_bootstrap.cc:492] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:30.574443 1203 tablet_bootstrap.cc:654] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:30.585577 713 leader_election.cc:343] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 17550eb0ee6b476f8a8c304451f678b2
I20251214 20:52:30.536829 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 18e119bc44874aecb4e83c8d6040c8cd (table default.Table_cf7dd9af89cc4f06adec7de4838b9a32 [id=466b9ab8c5ce4b04974f3bc7816087b1]) successfully deleted
W20251214 20:52:30.586524 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 49721.1us
I20251214 20:52:30.587559 1856 ts_tablet_manager.cc:1929] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.548267 848 leader_election.cc:304] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:30.593701 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 45549.9us
I20251214 20:52:30.593926 2712 raft_consensus.cc:2749] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:30.596164 712 leader_election.cc:343] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 17550eb0ee6b476f8a8c304451f678b2
I20251214 20:52:30.596295 712 leader_election.cc:304] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:30.598964 2747 raft_consensus.cc:2749] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.599252 1856 log.cc:1199] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/3d949abef1cb41a9905482d81a98d2c1
I20251214 20:52:30.599623 2738 tablet.cc:1722] T 738668adf2874d6894db6fa875f6990c P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:30.599788 1856 ts_tablet_manager.cc:1950] T 3d949abef1cb41a9905482d81a98d2c1 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:30.601321 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 3d949abef1cb41a9905482d81a98d2c1 (table default.Table_18ac89da14c642398e5f36730d316fee [id=56cb939dcaf243a389548a1d68d52081]) successfully deleted
W20251214 20:52:30.605365 442 catalog_manager.cc:4952] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 98b7cac3930a4a9eb16ac413c8f1e092 because the tablet was not found. No further retry: Not found: Tablet not found: 98b7cac3930a4a9eb16ac413c8f1e092
I20251214 20:52:30.605897 1856 tablet_replica.cc:333] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:30.606500 1856 raft_consensus.cc:2243] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.608341 1856 raft_consensus.cc:2272] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.609877 1856 ts_tablet_manager.cc:1916] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.612185 2749 catalog_manager.cc:1269] Loaded cluster ID: c3f04138608d4adc935bb66e1a72f2bb
I20251214 20:52:30.612365 2712 raft_consensus.cc:493] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.618026 1202 tablet_bootstrap.cc:492] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
W20251214 20:52:30.618058 851 consensus_peers.cc:597] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d949abef1cb41a9905482d81a98d2c1. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:30.609225 1203 tablet_bootstrap.cc:492] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:30.618216 1202 ts_tablet_manager.cc:1403] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.307s user 0.036s sys 0.040s
I20251214 20:52:30.618356 1203 ts_tablet_manager.cc:1403] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.004s
I20251214 20:52:30.618462 2712 raft_consensus.cc:515] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.619184 1203 raft_consensus.cc:359] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.619688 1203 raft_consensus.cc:385] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.619832 1203 raft_consensus.cc:740] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.620263 1203 consensus_queue.cc:260] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.621994 2712 leader_election.cc:290] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:30.622805 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6bdfbe4d15345de8be5de00bfe4ca07" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:30.623467 848 leader_election.cc:343] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.626444 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6bdfbe4d15345de8be5de00bfe4ca07" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:30.626735 1051 raft_consensus.cc:2468] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:30.626178 1203 ts_tablet_manager.cc:1434] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251214 20:52:30.627491 851 leader_election.cc:304] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:30.628036 2656 raft_consensus.cc:2804] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:30.628113 2656 raft_consensus.cc:493] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:30.628170 2656 raft_consensus.cc:3060] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:30.631635 1203 tablet_bootstrap.cc:492] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:30.634332 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_bdfed01cc4cd4458be399de18a738f45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.634881 451 master_service.cc:954] ReplaceTablet: received request to replace tablet 18e95ff0e0da4c7680f594ba31cdc3a1 from {username='slave'} at 127.0.0.1:50088
W20251214 20:52:30.634991 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdfed01cc4cd4458be399de18a738f45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.635550 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_58bb2f3098084d73bafbd51cde268a21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.635958 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58bb2f3098084d73bafbd51cde268a21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:30.636283 449 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1dbaa39efa04086a22cc3908ed691d4 consensus state has no leader
I20251214 20:52:30.637943 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_66472fe7b4dd4e3eb07839fe78a7bead" } new_table_name: "default.Table_ec8c8f9185be4470a6580c4e141e2227" modify_external_catalogs: true
I20251214 20:52:30.639979 1203 tablet_bootstrap.cc:654] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.641933 2749 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: loading cluster ID for follower catalog manager: success
I20251214 20:52:30.643566 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_baed953978784a8eb83ffe465a406dfb" } modify_external_catalogs: true
I20251214 20:52:30.643772 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_baed953978784a8eb83ffe465a406dfb" } modify_external_catalogs: true
I20251214 20:52:30.647862 1202 raft_consensus.cc:359] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.648106 1202 raft_consensus.cc:385] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.648165 1202 raft_consensus.cc:740] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.648394 1202 consensus_queue.cc:260] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.648821 1202 ts_tablet_manager.cc:1434] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20251214 20:52:30.649180 1202 tablet_bootstrap.cc:492] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:30.633498 2330 ts_manager.cc:194] Registered new tserver with Master: a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:30.660800 1203 tablet_bootstrap.cc:492] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:30.661113 1203 ts_tablet_manager.cc:1403] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20251214 20:52:30.662066 1203 raft_consensus.cc:359] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.662345 1203 raft_consensus.cc:385] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.662451 1203 raft_consensus.cc:740] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.662806 1203 consensus_queue.cc:260] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.671175 1203 ts_tablet_manager.cc:1434] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251214 20:52:30.673007 1856 ts_tablet_manager.cc:1929] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.683894 1856 log.cc:1199] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/f84e9dfc23d34eb0a379e8ac5a442e59
I20251214 20:52:30.686852 1856 ts_tablet_manager.cc:1950] T f84e9dfc23d34eb0a379e8ac5a442e59 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:30.687958 1977 ts_tablet_manager.cc:1929] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:30.688086 1977 log.cc:1199] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/47e813373fbd482298312ead4f037c01
I20251214 20:52:30.677639 1203 tablet_bootstrap.cc:492] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:30.690272 1203 tablet_bootstrap.cc:654] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.690457 1977 ts_tablet_manager.cc:1950] T 47e813373fbd482298312ead4f037c01 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:30.691354 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 426) took 2523 ms. Trace:
I20251214 20:52:30.691457 1977 rpcz_store.cc:276] 1214 20:52:28.167802 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.168570 (+ 768us) service_pool.cc:225] Handling call
1214 20:52:30.516633 (+2348063us) ts_tablet_manager.cc:1151] Deleting tablet 47e813373fbd482298312ead4f037c01
1214 20:52:30.516643 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.650859 (+134216us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.687793 (+ 36934us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.691328 (+ 3535us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2347315}
I20251214 20:52:30.691908 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 427) took 2524 ms. Trace:
I20251214 20:52:30.691999 1977 rpcz_store.cc:276] 1214 20:52:28.167917 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.168758 (+ 841us) service_pool.cc:225] Handling call
1214 20:52:30.691799 (+2523041us) ts_tablet_manager.cc:1151] Deleting tablet ebb26a5ae7224e9ca8a513c7b5013618
1214 20:52:30.691809 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.691893 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2522915}
I20251214 20:52:30.692266 1977 tablet_replica.cc:333] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:30.692452 1977 raft_consensus.cc:2243] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.692440 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet f84e9dfc23d34eb0a379e8ac5a442e59 (table default.Table_44bb60b4a8ec45329622c851c90b5aa8 [id=1fb24606cfcd4445800f51b30b7f0fdf]) successfully deleted
I20251214 20:52:30.692591 1977 raft_consensus.cc:2272] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.693104 1977 ts_tablet_manager.cc:1916] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.693477 1856 tablet_replica.cc:333] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
W20251214 20:52:30.693650 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e005feeb95b48b9ad33612ab81d5bce has no consensus state
I20251214 20:52:30.694272 1856 raft_consensus.cc:2243] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:30.694429 1856 raft_consensus.cc:2272] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.694388 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_d050172b06964a17a220a95fb84d1081"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.694913 1856 ts_tablet_manager.cc:1916] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:30.695030 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d050172b06964a17a220a95fb84d1081 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.699313 454 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_5de6830aaec645f2992e4cfaceb20c27 [id=9553b46856c94003a293135c814a7df7]). Expected version 2 got 0
I20251214 20:52:30.699579 2703 raft_consensus.cc:493] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.699744 2703 raft_consensus.cc:515] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.700057 454 catalog_manager.cc:5654] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:30.700433 2703 leader_election.cc:290] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:30.701699 454 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (table default.Table_fcafcf3c6ddb4457aea41320f7bf0b74 [id=4329d01e7f6147d0adad53fa86369b17]). Expected version 1 got 0
I20251214 20:52:30.704248 747 tablet_service.cc:1505] Processing CreateTablet for tablet 5b1576eff7e94f02a373e02f8e4a961d (DEFAULT_TABLE table=default.Table_529fb2317e754462b7ae49d9304e5402 [id=9187b59d5c7949169b104f586e7edad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.704248 752 tablet_service.cc:1505] Processing CreateTablet for tablet f8df264ec35c4a9d8a23f234d8c56bc6 (DEFAULT_TABLE table=default.Table_42e9fda7b7d14640a5804197bfb41683 [id=3bd3b6b8ee1f4d7abc6f888fc9d9004a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.704469 746 tablet_service.cc:1505] Processing CreateTablet for tablet e8c4693bad7a485ea769ab075c536b35 (DEFAULT_TABLE table=default.Table_a54f22962823421ea5ab3033e0d9add0 [id=5db835c4244847e4969597e99df50099]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.704694 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8df264ec35c4a9d8a23f234d8c56bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.704892 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c4693bad7a485ea769ab075c536b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.705469 1018 tablet_service.cc:1505] Processing CreateTablet for tablet f8df264ec35c4a9d8a23f234d8c56bc6 (DEFAULT_TABLE table=default.Table_42e9fda7b7d14640a5804197bfb41683 [id=3bd3b6b8ee1f4d7abc6f888fc9d9004a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.705616 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 5b1576eff7e94f02a373e02f8e4a961d (DEFAULT_TABLE table=default.Table_529fb2317e754462b7ae49d9304e5402 [id=9187b59d5c7949169b104f586e7edad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.705902 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8df264ec35c4a9d8a23f234d8c56bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.701489 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6a8863ace4541aea7463af24db27142" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:30.707541 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1576eff7e94f02a373e02f8e4a961d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.708102 1202 tablet_bootstrap.cc:654] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.711490 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1576eff7e94f02a373e02f8e4a961d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:30.712133 991 leader_election.cc:343] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.712556 447 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet e90d11d69b8d4fd49150aa67a8190e5b
I20251214 20:52:30.712802 1033 tablet_service.cc:1505] Processing CreateTablet for tablet e8c4693bad7a485ea769ab075c536b35 (DEFAULT_TABLE table=default.Table_a54f22962823421ea5ab3033e0d9add0 [id=5db835c4244847e4969597e99df50099]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.713197 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c4693bad7a485ea769ab075c536b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.714274 447 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_2d61f26706854e2cbf1555c7ecd45b2d" } new_table_name: "default.Table_5cffb2d16a4847caa5cf6a2c4c238da8" modify_external_catalogs: true
I20251214 20:52:30.715755 2703 raft_consensus.cc:493] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.715938 2703 raft_consensus.cc:515] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.716470 452 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_eaeae6c30719483681416f2d216d1209"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.716590 2703 leader_election.cc:290] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:30.716969 452 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaeae6c30719483681416f2d216d1209 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.718022 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1f4ea72c6274569a98cab6afdc55d55" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:30.712478 450 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778] alter complete (version 1)
W20251214 20:52:30.718616 989 leader_election.cc:343] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.703980 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_98b0ed640a93417ba49359781348b105"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.719441 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98b0ed640a93417ba49359781348b105 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.726248 1203 tablet_bootstrap.cc:492] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:30.726540 1203 ts_tablet_manager.cc:1403] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20251214 20:52:30.726909 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 47e813373fbd482298312ead4f037c01 (table default.Table_a60363e46077464b968f9187a8c7a8fc [id=708908b1be53473f8c5f7e191128e7e2]) successfully deleted
W20251214 20:52:30.727425 441 catalog_manager.cc:4952] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet ebb26a5ae7224e9ca8a513c7b5013618 because the tablet was not found. No further retry: Not found: Tablet not found: ebb26a5ae7224e9ca8a513c7b5013618
I20251214 20:52:30.727689 1203 raft_consensus.cc:359] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.727916 1203 raft_consensus.cc:385] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.728003 1203 raft_consensus.cc:740] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.728245 1203 consensus_queue.cc:260] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.729323 893 tablet_service.cc:1505] Processing CreateTablet for tablet f8df264ec35c4a9d8a23f234d8c56bc6 (DEFAULT_TABLE table=default.Table_42e9fda7b7d14640a5804197bfb41683 [id=3bd3b6b8ee1f4d7abc6f888fc9d9004a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.729717 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8df264ec35c4a9d8a23f234d8c56bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.730355 1203 ts_tablet_manager.cc:1434] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:30.730741 1203 tablet_bootstrap.cc:492] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:30.732118 743 tablet_service.cc:1505] Processing CreateTablet for tablet 03ab5235046c4ec8afc4bc6730d6cfa8 (DEFAULT_TABLE table=default.Table_cca1629ea2974eddb3a2fab0ae9f4500 [id=7ceb70a9b9b0458692720cd351d9a3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.732559 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03ab5235046c4ec8afc4bc6730d6cfa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.732638 1203 tablet_bootstrap.cc:654] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.732558 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 89a0435db9a1410db060a91c18ae1c5e (DEFAULT_TABLE table=default.Table_f8b17194060c4627a7770e45799e082b [id=1c9da7e7dc974945b2b5ccf304d1c1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.732705 451 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_26fdf6b490ab4f6cb488f406406a190c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.733253 451 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26fdf6b490ab4f6cb488f406406a190c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.733444 756 tablet_service.cc:1505] Processing CreateTablet for tablet 89a0435db9a1410db060a91c18ae1c5e (DEFAULT_TABLE table=default.Table_f8b17194060c4627a7770e45799e082b [id=1c9da7e7dc974945b2b5ccf304d1c1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.733758 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a0435db9a1410db060a91c18ae1c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.733973 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 9cc05b7f5c114604afa025338b6fa617 (DEFAULT_TABLE table=default.Table_b3dbc2e42cc64d9d96c48ecc1b474480 [id=e78b07fcb7a341fdbd0c244409c3c445]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.734887 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 03ab5235046c4ec8afc4bc6730d6cfa8 (DEFAULT_TABLE table=default.Table_cca1629ea2974eddb3a2fab0ae9f4500 [id=7ceb70a9b9b0458692720cd351d9a3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.735394 1021 tablet_service.cc:1505] Processing CreateTablet for tablet 636c209967fd4fe78d91d9e7a6fbf613 (DEFAULT_TABLE table=default.Table_62fb3a07f4a04029a6080d9d0e35d151 [id=c2886db5a22c43b5b371935e9f47187a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.735809 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 259762515ce44ca6b7f4cd855310d1ec (DEFAULT_TABLE table=default.Table_e7a205677f614917bc69ca73b5fb54da [id=262d9252040b4138ad42d1c05a402cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.736194 1022 tablet_service.cc:1505] Processing CreateTablet for tablet 0b7dd0ab78534a34ba342ec6e4fae129 (DEFAULT_TABLE table=default.Table_4f6c749899d64d8884fd2ac83cfb13fd [id=2be253c680464ec4b0d417fbf2b45761]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.736647 2703 raft_consensus.cc:493] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.736804 2703 raft_consensus.cc:515] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.737079 876 tablet_service.cc:1505] Processing CreateTablet for tablet 5b1576eff7e94f02a373e02f8e4a961d (DEFAULT_TABLE table=default.Table_529fb2317e754462b7ae49d9304e5402 [id=9187b59d5c7949169b104f586e7edad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.737435 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1576eff7e94f02a373e02f8e4a961d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.737553 2703 leader_election.cc:290] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:30.738400 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a000951f61948ac85fdd431f3d162c2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:30.738579 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 1afeb64843e64f5b9b45259b1a3ec3d8 (DEFAULT_TABLE table=default.Table_1dc40537749846a8a77948884d646257 [id=ae41e2897b3a4deda0dacb28baa72adf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:30.739609 991 leader_election.cc:343] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.740240 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 2ef81c05366743d28579465e95a79293 (DEFAULT_TABLE table=default.Table_e2a035220b924122a538db32c77a683b [id=f97b0be6362b47b49c15807bdf090e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.740762 755 tablet_service.cc:1505] Processing CreateTablet for tablet 2ef81c05366743d28579465e95a79293 (DEFAULT_TABLE table=default.Table_e2a035220b924122a538db32c77a683b [id=f97b0be6362b47b49c15807bdf090e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.741150 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet e90d11d69b8d4fd49150aa67a8190e5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.743290 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 5f6be4dbadc74f468fe5f92ecf158e63 (DEFAULT_TABLE table=default.Table_47682ee6f46147558b5033379c2fe733 [id=788e2409150b408996fcc9e70c026bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.745537 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 557fcbbc7889434e87ca82177c5d0eb0 (DEFAULT_TABLE table=default.Table_7b56d13e748e405a821ea513405846fa [id=2e995b9d47b04d3d8f5a01482c57d0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.747027 1036 tablet_service.cc:1505] Processing CreateTablet for tablet c75e6701210e4d688a96678bf042860a (DEFAULT_TABLE table=default.Table_040e46c43e8446f48a60e34f0892bf57 [id=2b6d1f218393414d868d37d390b35b58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.747720 2703 raft_consensus.cc:493] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.747877 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 6e005feeb95b48b9ad33612ab81d5bce (DEFAULT_TABLE table=default.Table_ec8c8f9185be4470a6580c4e141e2227 [id=8872bfac95c64cbba1ccb84d1d9f2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.747453 2749 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: acquiring CA information for follower catalog manager: success
I20251214 20:52:30.747890 2703 raft_consensus.cc:515] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:30.748920 2703 leader_election.cc:290] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:30.749869 886 tablet_service.cc:1505] Processing CreateTablet for tablet 03ab5235046c4ec8afc4bc6730d6cfa8 (DEFAULT_TABLE table=default.Table_cca1629ea2974eddb3a2fab0ae9f4500 [id=7ceb70a9b9b0458692720cd351d9a3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.749926 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d157ac8f74eb7911df05ab5e4a3d5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:30.750550 991 leader_election.cc:343] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.750942 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03ab5235046c4ec8afc4bc6730d6cfa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.751519 895 tablet_service.cc:1505] Processing CreateTablet for tablet 636c209967fd4fe78d91d9e7a6fbf613 (DEFAULT_TABLE table=default.Table_62fb3a07f4a04029a6080d9d0e35d151 [id=c2886db5a22c43b5b371935e9f47187a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.751878 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 80e28c4d87774256acb8e9638ded8689 (DEFAULT_TABLE table=default.Table_eec6b34ea20f4ef096ef8df379a3bddb [id=05a65b055df144b0bd3e83d8baba49a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.752197 887 tablet_service.cc:1552] Processing DeleteTablet for tablet e90d11d69b8d4fd49150aa67a8190e5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:30.752228 885 tablet_service.cc:1505] Processing CreateTablet for tablet 259762515ce44ca6b7f4cd855310d1ec (DEFAULT_TABLE table=default.Table_e7a205677f614917bc69ca73b5fb54da [id=262d9252040b4138ad42d1c05a402cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.752600 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259762515ce44ca6b7f4cd855310d1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.753063 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 07e5cf36a7cc476fafd3473a1de9eced (DEFAULT_TABLE table=default.Table_40e2218a9f58420e9a09ccfbda122485 [id=9fa32d9e7b7c49baa67e36e0eb996e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.753325 879 tablet_service.cc:1505] Processing CreateTablet for tablet 9cc05b7f5c114604afa025338b6fa617 (DEFAULT_TABLE table=default.Table_b3dbc2e42cc64d9d96c48ecc1b474480 [id=e78b07fcb7a341fdbd0c244409c3c445]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.753901 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc05b7f5c114604afa025338b6fa617. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.754235 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 91798ce3e88e4f82b59c52222dd83aef (DEFAULT_TABLE table=default.Table_8cd3b56154ec4d9fac4ed571a2287cfa [id=39df4da3b39545d3ba2c9345618ae532]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.755316 891 tablet_service.cc:1505] Processing CreateTablet for tablet e8c4693bad7a485ea769ab075c536b35 (DEFAULT_TABLE table=default.Table_a54f22962823421ea5ab3033e0d9add0 [id=5db835c4244847e4969597e99df50099]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.755393 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 243528f1aeac425c8be47f9fae802f52 (DEFAULT_TABLE table=default.Table_8923490b824a47139d8f739d65e4845b [id=2ffffc8a53e04583a2f93b287fe083bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.755810 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8c4693bad7a485ea769ab075c536b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.756775 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 74d2f8e5481d40b39cc4df34b0d4f819 (DEFAULT_TABLE table=default.Table_0fb8f39742d245ac9d20fc7d6bf7e7c3 [id=552c8b213ba64d4bbdd3363cdc6625ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.758752 880 tablet_service.cc:1505] Processing CreateTablet for tablet 1afeb64843e64f5b9b45259b1a3ec3d8 (DEFAULT_TABLE table=default.Table_1dc40537749846a8a77948884d646257 [id=ae41e2897b3a4deda0dacb28baa72adf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.759179 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1afeb64843e64f5b9b45259b1a3ec3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.759582 894 tablet_service.cc:1505] Processing CreateTablet for tablet 89a0435db9a1410db060a91c18ae1c5e (DEFAULT_TABLE table=default.Table_f8b17194060c4627a7770e45799e082b [id=1c9da7e7dc974945b2b5ccf304d1c1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.759969 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 636c209967fd4fe78d91d9e7a6fbf613. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.760537 877 tablet_service.cc:1505] Processing CreateTablet for tablet 2ef81c05366743d28579465e95a79293 (DEFAULT_TABLE table=default.Table_e2a035220b924122a538db32c77a683b [id=f97b0be6362b47b49c15807bdf090e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.760865 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef81c05366743d28579465e95a79293. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.753070 887 tablet_service.cc:1505] Processing CreateTablet for tablet 0b7dd0ab78534a34ba342ec6e4fae129 (DEFAULT_TABLE table=default.Table_4f6c749899d64d8884fd2ac83cfb13fd [id=2be253c680464ec4b0d417fbf2b45761]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.763316 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7dd0ab78534a34ba342ec6e4fae129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.764175 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1f4ea72c6274569a98cab6afdc55d55" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:30.765271 991 leader_election.cc:343] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.765435 991 leader_election.cc:304] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:30.765908 2703 raft_consensus.cc:2749] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.766793 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc05b7f5c114604afa025338b6fa617. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.767326 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a0435db9a1410db060a91c18ae1c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.767550 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243528f1aeac425c8be47f9fae802f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.767796 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef81c05366743d28579465e95a79293. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.768008 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80e28c4d87774256acb8e9638ded8689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.768380 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1afeb64843e64f5b9b45259b1a3ec3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.768575 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259762515ce44ca6b7f4cd855310d1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.768760 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d2f8e5481d40b39cc4df34b0d4f819. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.768941 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03ab5235046c4ec8afc4bc6730d6cfa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.769310 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75e6701210e4d688a96678bf042860a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.769503 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f6be4dbadc74f468fe5f92ecf158e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.769876 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91798ce3e88e4f82b59c52222dd83aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.770256 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e5cf36a7cc476fafd3473a1de9eced. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.770458 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557fcbbc7889434e87ca82177c5d0eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.770767 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e005feeb95b48b9ad33612ab81d5bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.771762 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7dd0ab78534a34ba342ec6e4fae129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.775300 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 636c209967fd4fe78d91d9e7a6fbf613. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.779796 888 tablet_service.cc:1505] Processing CreateTablet for tablet 5f6be4dbadc74f468fe5f92ecf158e63 (DEFAULT_TABLE table=default.Table_47682ee6f46147558b5033379c2fe733 [id=788e2409150b408996fcc9e70c026bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.780223 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f6be4dbadc74f468fe5f92ecf158e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.741130 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef81c05366743d28579465e95a79293. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.741243 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a000951f61948ac85fdd431f3d162c2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:30.744797 742 tablet_service.cc:1505] Processing CreateTablet for tablet 557fcbbc7889434e87ca82177c5d0eb0 (DEFAULT_TABLE table=default.Table_7b56d13e748e405a821ea513405846fa [id=2e995b9d47b04d3d8f5a01482c57d0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.746539 749 tablet_service.cc:1505] Processing CreateTablet for tablet c75e6701210e4d688a96678bf042860a (DEFAULT_TABLE table=default.Table_040e46c43e8446f48a60e34f0892bf57 [id=2b6d1f218393414d868d37d390b35b58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.749536 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d157ac8f74eb7911df05ab5e4a3d5" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:30.796075 757 tablet_service.cc:1505] Processing CreateTablet for tablet 6e005feeb95b48b9ad33612ab81d5bce (DEFAULT_TABLE table=default.Table_ec8c8f9185be4470a6580c4e141e2227 [id=8872bfac95c64cbba1ccb84d1d9f2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.740826 750 tablet_service.cc:1505] Processing CreateTablet for tablet 1afeb64843e64f5b9b45259b1a3ec3d8 (DEFAULT_TABLE table=default.Table_1dc40537749846a8a77948884d646257 [id=ae41e2897b3a4deda0dacb28baa72adf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.784511 754 tablet_service.cc:1505] Processing CreateTablet for tablet 5f6be4dbadc74f468fe5f92ecf158e63 (DEFAULT_TABLE table=default.Table_47682ee6f46147558b5033379c2fe733 [id=788e2409150b408996fcc9e70c026bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.784706 741 tablet_service.cc:1505] Processing CreateTablet for tablet 259762515ce44ca6b7f4cd855310d1ec (DEFAULT_TABLE table=default.Table_e7a205677f614917bc69ca73b5fb54da [id=262d9252040b4138ad42d1c05a402cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.796533 738 tablet_service.cc:1505] Processing CreateTablet for tablet 91798ce3e88e4f82b59c52222dd83aef (DEFAULT_TABLE table=default.Table_8cd3b56154ec4d9fac4ed571a2287cfa [id=39df4da3b39545d3ba2c9345618ae532]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.784845 740 tablet_service.cc:1505] Processing CreateTablet for tablet 0b7dd0ab78534a34ba342ec6e4fae129 (DEFAULT_TABLE table=default.Table_4f6c749899d64d8884fd2ac83cfb13fd [id=2be253c680464ec4b0d417fbf2b45761]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.783073 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6a8863ace4541aea7463af24db27142" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:30.796984 989 leader_election.cc:343] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.797135 989 leader_election.cc:304] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:30.797562 989 leader_election.cc:343] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.797669 989 leader_election.cc:304] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:30.798162 745 tablet_service.cc:1505] Processing CreateTablet for tablet 243528f1aeac425c8be47f9fae802f52 (DEFAULT_TABLE table=default.Table_8923490b824a47139d8f739d65e4845b [id=2ffffc8a53e04583a2f93b287fe083bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.798607 2735 raft_consensus.cc:2749] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.798804 2735 raft_consensus.cc:2749] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.799535 739 tablet_service.cc:1505] Processing CreateTablet for tablet 9cc05b7f5c114604afa025338b6fa617 (DEFAULT_TABLE table=default.Table_b3dbc2e42cc64d9d96c48ecc1b474480 [id=e78b07fcb7a341fdbd0c244409c3c445]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.801481 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e005feeb95b48b9ad33612ab81d5bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.801930 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75e6701210e4d688a96678bf042860a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.802171 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1afeb64843e64f5b9b45259b1a3ec3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.802346 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f6be4dbadc74f468fe5f92ecf158e63. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:30.802510 989 leader_election.cc:343] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.802652 989 leader_election.cc:304] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:30.802649 2772 tablet.cc:1722] T 9da5e98c6f4843449886176e9fcfa8e1 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:30.803897 2703 raft_consensus.cc:2749] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.807698 2730 tablet.cc:1722] T 9da5e98c6f4843449886176e9fcfa8e1 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:30.811614 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a0435db9a1410db060a91c18ae1c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.768983 2656 raft_consensus.cc:515] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.814827 2656 leader_election.cc:290] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:30.799223 751 tablet_service.cc:1505] Processing CreateTablet for tablet 636c209967fd4fe78d91d9e7a6fbf613 (DEFAULT_TABLE table=default.Table_62fb3a07f4a04029a6080d9d0e35d151 [id=c2886db5a22c43b5b371935e9f47187a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.818209 753 tablet_service.cc:1505] Processing CreateTablet for tablet 80e28c4d87774256acb8e9638ded8689 (DEFAULT_TABLE table=default.Table_eec6b34ea20f4ef096ef8df379a3bddb [id=05a65b055df144b0bd3e83d8baba49a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.818567 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6bdfbe4d15345de8be5de00bfe4ca07" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:30.818787 744 tablet_service.cc:1505] Processing CreateTablet for tablet 07e5cf36a7cc476fafd3473a1de9eced (DEFAULT_TABLE table=default.Table_40e2218a9f58420e9a09ccfbda122485 [id=9fa32d9e7b7c49baa67e36e0eb996e2e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:30.819525 848 leader_election.cc:343] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.819811 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259762515ce44ca6b7f4cd855310d1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.820010 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7dd0ab78534a34ba342ec6e4fae129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.820184 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc05b7f5c114604afa025338b6fa617. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.820734 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91798ce3e88e4f82b59c52222dd83aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.820935 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243528f1aeac425c8be47f9fae802f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.821056 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557fcbbc7889434e87ca82177c5d0eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.822547 1203 tablet_bootstrap.cc:492] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:30.822757 1203 ts_tablet_manager.cc:1403] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.000s
I20251214 20:52:30.825760 1203 raft_consensus.cc:359] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.826607 1203 raft_consensus.cc:385] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.827860 1203 raft_consensus.cc:740] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.829550 1203 consensus_queue.cc:260] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.830360 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e5cf36a7cc476fafd3473a1de9eced. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.834409 1977 ts_tablet_manager.cc:1929] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.834545 1977 log.cc:1199] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/453cf734f8b34b59aa246d618ffeade5
I20251214 20:52:30.834846 1977 ts_tablet_manager.cc:1950] T 453cf734f8b34b59aa246d618ffeade5 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:30.840394 885 tablet_service.cc:1505] Processing CreateTablet for tablet 557fcbbc7889434e87ca82177c5d0eb0 (DEFAULT_TABLE table=default.Table_7b56d13e748e405a821ea513405846fa [id=2e995b9d47b04d3d8f5a01482c57d0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.842041 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557fcbbc7889434e87ca82177c5d0eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.842118 2703 raft_consensus.cc:493] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.842276 2703 raft_consensus.cc:515] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.841643 1202 tablet_bootstrap.cc:492] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:30.842629 1202 ts_tablet_manager.cc:1403] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.194s user 0.001s sys 0.019s
I20251214 20:52:30.843031 2703 leader_election.cc:290] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:30.843557 1203 ts_tablet_manager.cc:1434] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251214 20:52:30.851549 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80e28c4d87774256acb8e9638ded8689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.854717 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 636c209967fd4fe78d91d9e7a6fbf613. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.861065 446 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 2f75657d7bc44f6393c55ab119eb7dc2
I20251214 20:52:30.862576 1203 tablet_bootstrap.cc:492] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:30.864686 1203 tablet_bootstrap.cc:654] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.867158 452 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_32323ff67ecd4fe38fa02169904b88bf" } new_table_name: "default.Table_25630f29e4904aaf86f8f65078bcf81e" modify_external_catalogs: true
I20251214 20:52:30.868592 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 348d8dcef97042279bc0d5ff31d3c137
W20251214 20:52:30.870669 447 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 438b8ec5e9834442994966c8d83c1a31 has no consensus state
I20251214 20:52:30.870970 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f75657d7bc44f6393c55ab119eb7dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.871498 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c1893b3761c49ceb7cd71b3bd9f2cb4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:30.876801 1202 raft_consensus.cc:359] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:30.877015 1202 raft_consensus.cc:385] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.877061 1202 raft_consensus.cc:740] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.845129 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c1893b3761c49ceb7cd71b3bd9f2cb4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:30.879388 989 leader_election.cc:343] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:30.880050 991 leader_election.cc:343] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.880141 991 leader_election.cc:304] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:30.880911 2735 raft_consensus.cc:2749] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:30.874943 889 tablet_service.cc:1505] Processing CreateTablet for tablet 6e005feeb95b48b9ad33612ab81d5bce (DEFAULT_TABLE table=default.Table_ec8c8f9185be4470a6580c4e141e2227 [id=8872bfac95c64cbba1ccb84d1d9f2483]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.875747 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 422) took 2708 ms. Trace:
I20251214 20:52:30.881500 1977 rpcz_store.cc:276] 1214 20:52:28.167334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.169119 (+ 1785us) service_pool.cc:225] Handling call
1214 20:52:30.692218 (+2523099us) ts_tablet_manager.cc:1151] Deleting tablet 453cf734f8b34b59aa246d618ffeade5
1214 20:52:30.692225 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.831386 (+139161us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.834303 (+ 2917us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.875718 (+ 41415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2522869}
I20251214 20:52:30.881904 1977 tablet_replica.cc:333] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:30.881981 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 348d8dcef97042279bc0d5ff31d3c137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.882560 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_9196cd658f9041269835197751563391" } new_table_name: "default.Table_196d0f488e0b42aabd518511a4289dc3" modify_external_catalogs: true
I20251214 20:52:30.884146 448 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 1def5f0de5ce4eb0bed36f626c4f0683
I20251214 20:52:30.886879 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 1def5f0de5ce4eb0bed36f626c4f0683 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:30.888787 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e005feeb95b48b9ad33612ab81d5bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.913954 884 tablet_service.cc:1505] Processing CreateTablet for tablet 80e28c4d87774256acb8e9638ded8689 (DEFAULT_TABLE table=default.Table_eec6b34ea20f4ef096ef8df379a3bddb [id=05a65b055df144b0bd3e83d8baba49a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.916910 892 tablet_service.cc:1505] Processing CreateTablet for tablet 07e5cf36a7cc476fafd3473a1de9eced (DEFAULT_TABLE table=default.Table_40e2218a9f58420e9a09ccfbda122485 [id=9fa32d9e7b7c49baa67e36e0eb996e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:30.918027 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e5cf36a7cc476fafd3473a1de9eced. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.920388 2749 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251214 20:52:30.930414 453 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_a5e4f40e603f4fd0a3d204f9c0298023"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.931450 453 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5e4f40e603f4fd0a3d204f9c0298023 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.935876 2703 raft_consensus.cc:493] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:30.936081 2703 raft_consensus.cc:515] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.936479 446 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:30.936759 2703 leader_election.cc:290] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:30.937774 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d424e3387698434f92e84eb1d43f5949" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:30.938357 989 leader_election.cc:343] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:30.938318 451 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_331c1dbc1c1f448e9f09c5fc935f9894" } new_table_name: "default.Table_27222912e9214b879d8a30975aba5688" modify_external_catalogs: true
I20251214 20:52:30.289966 2397 rpcz_store.cc:276] 1214 20:52:26.245117 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:26.245187 (+ 70us) service_pool.cc:225] Handling call
1214 20:52:29.022658 (+2777471us) raft_consensus.cc:1471] Updating replica for 299 ops
1214 20:52:29.033690 (+ 11032us) raft_consensus.cc:1534] Early marking committed up to index 0
1214 20:52:29.033741 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 299 ops
1214 20:52:29.040745 (+ 7004us) write_op.cc:270] Start()
1214 20:52:29.041167 (+ 422us) write_op.cc:276] Timestamp: P: 1765745537461528 usec, L: 0
1214 20:52:29.165523 (+124356us) write_op.cc:270] Start()
1214 20:52:29.165673 (+ 150us) write_op.cc:276] Timestamp: P: 1765745537487460 usec, L: 0
1214 20:52:29.166455 (+ 782us) write_op.cc:270] Start()
1214 20:52:29.166834 (+ 379us) write_op.cc:276] Timestamp: P: 1765745537500116 usec, L: 0
1214 20:52:29.167690 (+ 856us) write_op.cc:270] Start()
1214 20:52:29.167770 (+ 80us) write_op.cc:276] Timestamp: P: 1765745537503983 usec, L: 0
1214 20:52:29.168355 (+ 585us) write_op.cc:270] Start()
1214 20:52:29.168504 (+ 149us) write_op.cc:276] Timestamp: P: 1765745537635119 usec, L: 0
1214 20:52:29.169496 (+ 992us) write_op.cc:270] Start()
1214 20:52:29.169643 (+ 147us) write_op.cc:276] Timestamp: P: 1765745537804716 usec, L: 0
1214 20:52:29.171896 (+ 2253us) write_op.cc:270] Start()
1214 20:52:29.172795 (+ 899us) write_op.cc:276] Timestamp: P: 1765745537857660 usec, L: 0
1214 20:52:29.177228 (+ 4433us) write_op.cc:270] Start()
1214 20:52:29.177683 (+ 455us) write_op.cc:276] Timestamp: P: 1765745537933171 usec, L: 0
1214 20:52:29.182443 (+ 4760us) write_op.cc:270] Start()
1214 20:52:29.182734 (+ 291us) write_op.cc:276] Timestamp: P: 1765745538002530 usec, L: 0
1214 20:52:29.186964 (+ 4230us) write_op.cc:270] Start()
1214 20:52:29.187253 (+ 289us) write_op.cc:276] Timestamp: P: 1765745538055685 usec, L: 0
1214 20:52:29.191171 (+ 3918us) write_op.cc:270] Start()
1214 20:52:29.191542 (+ 371us) write_op.cc:276] Timestamp: P: 1765745538056374 usec, L: 0
1214 20:52:29.194883 (+ 3341us) write_op.cc:270] Start()
1214 20:52:29.195334 (+ 451us) write_op.cc:276] Timestamp: P: 1765745538179515 usec, L: 0
1214 20:52:29.199072 (+ 3738us) write_op.cc:270] Start()
1214 20:52:29.199461 (+ 389us) write_op.cc:276] Timestamp: P: 1765745538211660 usec, L: 0
1214 20:52:29.202769 (+ 3308us) write_op.cc:270] Start()
1214 20:52:29.203202 (+ 433us) write_op.cc:276] Timestamp: P: 1765745538243079 usec, L: 0
1214 20:52:29.206972 (+ 3770us) write_op.cc:270] Start()
1214 20:52:29.207333 (+ 361us) write_op.cc:276] Timestamp: P: 1765745538257284 usec, L: 0
1214 20:52:29.210487 (+ 3154us) write_op.cc:270] Start()
1214 20:52:29.210847 (+ 360us) write_op.cc:276] Timestamp: P: 1765745538285823 usec, L: 0
1214 20:52:29.214860 (+ 4013us) write_op.cc:270] Start()
1214 20:52:29.215531 (+ 671us) write_op.cc:276] Timestamp: P: 1765745538311267 usec, L: 0
1214 20:52:29.219243 (+ 3712us) write_op.cc:270] Start()
1214 20:52:29.219614 (+ 371us) write_op.cc:276] Timestamp: P: 1765745538314348 usec, L: 0
1214 20:52:29.223692 (+ 4078us) write_op.cc:270] Start()
1214 20:52:29.224490 (+ 798us) write_op.cc:276] Timestamp: P: 1765745538315851 usec, L: 0
1214 20:52:29.227861 (+ 3371us) write_op.cc:270] Start()
1214 20:52:29.228153 (+ 292us) write_op.cc:276] Timestamp: P: 1765745538351239 usec, L: 0
1214 20:52:29.231000 (+ 2847us) write_op.cc:270] Start()
1214 20:52:29.231843 (+ 843us) write_op.cc:276] Timestamp: P: 1765745538388388 usec, L: 0
1214 20:52:29.236059 (+ 4216us) write_op.cc:270] Start()
1214 20:52:29.236788 (+ 729us) write_op.cc:276] Timestamp: P: 1765745538653121 usec, L: 0
1214 20:52:29.241220 (+ 4432us) write_op.cc:270] Start()
1214 20:52:29.241790 (+ 570us) write_op.cc:276] Timestamp: P: 1765745538664062 usec, L: 0
1214 20:52:29.245073 (+ 3283us) write_op.cc:270] Start()
1214 20:52:29.245650 (+ 577us) write_op.cc:276] Timestamp: P: 1765745538667743 usec, L: 0
1214 20:52:29.249844 (+ 4194us) write_op.cc:270] Start()
1214 20:52:29.250153 (+ 309us) write_op.cc:276] Timestamp: P: 1765745538678174 usec, L: 0
1214 20:52:29.252357 (+ 2204us) write_op.cc:270] Start()
1214 20:52:29.252594 (+ 237us) write_op.cc:276] Timestamp: P: 1765745538686997 usec, L: 0
1214 20:52:29.254359 (+ 1765us) write_op.cc:270] Start()
1214 20:52:29.254776 (+ 417us) write_op.cc:276] Timestamp: P: 1765745538692180 usec, L: 0
1214 20:52:29.256265 (+ 1489us) write_op.cc:270] Start()
1214 20:52:29.256571 (+ 306us) write_op.cc:276] Timestamp: P: 1765745538700701 usec, L: 0
1214 20:52:29.259267 (+ 2696us) write_op.cc:270] Start()
1214 20:52:29.288376 (+ 29109us) write_op.cc:276] Timestamp: P: 1765745538708457 usec, L: 0
1214 20:52:29.288675 (+ 299us) write_op.cc:270] Start()
1214 20:52:29.288707 (+ 32us) write_op.cc:276] Timestamp: P: 1765745538710592 usec, L: 0
1214 20:52:29.288885 (+ 178us) write_op.cc:270] Start()
1214 20:52:29.288905 (+ 20us) write_op.cc:276] Timestamp: P: 1765745538711739 usec, L: 0
1214 20:52:29.300410 (+ 11505us) write_op.cc:270] Start()
1214 20:52:29.300484 (+ 74us) write_op.cc:276] Timestamp: P: 1765745538712899 usec, L: 0
1214 20:52:29.300737 (+ 253us) write_op.cc:270] Start()
1214 20:52:29.300761 (+ 24us) write_op.cc:276] Timestamp: P: 1765745538714644 usec, L: 0
1214 20:52:29.300915 (+ 154us) write_op.cc:270] Start()
1214 20:52:29.300933 (+ 18us) write_op.cc:276] Timestamp: P: 1765745538717726 usec, L: 0
1214 20:52:29.301060 (+ 127us) write_op.cc:270] Start()
1214 20:52:29.301080 (+ 20us) write_op.cc:276] Timestamp: P: 1765745538726575 usec, L: 0
1214 20:52:29.301208 (+ 128us) write_op.cc:270] Start()
1214 20:52:29.301225 (+ 17us) write_op.cc:276] Timestamp: P: 1765745538739114 usec, L: 0
1214 20:52:29.309144 (+ 7919us) write_op.cc:270] Start()
1214 20:52:29.309195 (+ 51us) write_op.cc:276] Timestamp: P: 1765745538864555 usec, L: 0
1214 20:52:29.309399 (+ 204us) write_op.cc:270] Start()
1214 20:52:29.309422 (+ 23us) write_op.cc:276] Timestamp: P: 1765745538872037 usec, L: 0
1214 20:52:29.310596 (+ 1174us) write_op.cc:270] Start()
1214 20:52:29.310621 (+ 25us) write_op.cc:276] Timestamp: P: 1765745538874162 usec, L: 0
1214 20:52:29.310770 (+ 149us) write_op.cc:270] Start()
1214 20:52:29.310791 (+ 21us) write_op.cc:276] Timestamp: P: 1765745538882214 usec, L: 0
1214 20:52:29.310943 (+ 152us) write_op.cc:270] Start()
1214 20:52:29.310963 (+ 20us) write_op.cc:276] Timestamp: P: 1765745538883088 usec, L: 0
1214 20:52:29.311094 (+ 131us) write_op.cc:270] Start()
1214 20:52:29.311113 (+ 19us) write_op.cc:276] Timestamp: P: 1765745538886982 usec, L: 0
1214 20:52:29.311277 (+ 164us) write_op.cc:270] Start()
1214 20:52:29.311297 (+ 20us) write_op.cc:276] Timestamp: P: 1765745538888506 usec, L: 0
1214 20:52:29.311415 (+ 118us) write_op.cc:270] Start()
1214 20:52:29.311432 (+ 17us) write_op.cc:276] Timestamp: P: 1765745539100864 usec, L: 0
1214 20:52:29.311551 (+ 119us) write_op.cc:270] Start()
1214 20:52:29.311571 (+ 20us) write_op.cc:276] Timestamp: P: 1765745539109147 usec, L: 0
1214 20:52:29.311710 (+ 139us) write_op.cc:270] Start()
1214 20:52:29.311728 (+ 18us) write_op.cc:276] Timestamp: P: 1765745539109795 usec, L: 0
1214 20:52:29.311849 (+ 121us) write_op.cc:270] Start()
1214 20:52:29.311878 (+ 29us) write_op.cc:276] Timestamp: P: 1765745539121164 usec, L: 0
1214 20:52:29.312043 (+ 165us) write_op.cc:270] Start()
1214 20:52:29.312063 (+ 20us) write_op.cc:276] Timestamp: P: 1765745539151688 usec, L: 0
1214 20:52:29.314171 (+ 2108us) write_op.cc:270] Start()
1214 20:52:29.314193 (+ 22us) write_op.cc:276] Timestamp: P: 1765745539235875 usec, L: 0
1214 20:52:29.314379 (+ 186us) write_op.cc:270] Start()
1214 20:52:29.314397 (+ 18us) write_op.cc:276] Timestamp: P: 1765745539256527 usec, L: 0
1214 20:52:29.314571 (+ 174us) write_op.cc:270] Start()
1214 20:52:29.314588 (+ 17us) write_op.cc:276] Timestamp: P: 1765745539292274 usec, L: 0
1214 20:52:29.314701 (+ 113us) write_op.cc:270] Start()
1214 20:52:29.314717 (+ 16us) write_op.cc:276] Timestamp: P: 1765745539661558 usec, L: 0
1214 20:52:29.314809 (+ 92us) write_op.cc:270] Start()
1214 20:52:29.314824 (+ 15us) write_op.cc:276] Timestamp: P: 1765745539873229 usec, L: 0
1214 20:52:29.314953 (+ 129us) write_op.cc:270] Start()
1214 20:52:29.315062 (+ 109us) write_op.cc:276] Timestamp: P: 1765745539919991 usec, L: 0
1214 20:52:29.315226 (+ 164us) write_op.cc:270] Start()
1214 20:52:29.315245 (+ 19us) write_op.cc:276] Timestamp: P: 1765745539976327 usec, L: 0
1214 20:52:29.315360 (+ 115us) write_op.cc:270] Start()
1214 20:52:29.315377 (+ 17us) write_op.cc:276] Timestamp: P: 1765745539984247 usec, L: 0
1214 20:52:29.315499 (+ 122us) write_op.cc:270] Start()
1214 20:52:29.315517 (+ 18us) write_op.cc:276] Timestamp: P: 1765745539991469 usec, L: 0
1214 20:52:29.315638 (+ 121us) write_op.cc:270] Start()
1214 20:52:29.315665 (+ 27us) write_op.cc:276] Timestamp: P: 1765745540007907 usec, L: 0
1214 20:52:29.315774 (+ 109us) write_op.cc:270] Start()
1214 20:52:29.315791 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540015636 usec, L: 0
1214 20:52:29.315932 (+ 141us) write_op.cc:270] Start()
1214 20:52:29.315974 (+ 42us) write_op.cc:276] Timestamp: P: 1765745540023953 usec, L: 0
1214 20:52:29.316081 (+ 107us) write_op.cc:270] Start()
1214 20:52:29.316098 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540036261 usec, L: 0
1214 20:52:29.316209 (+ 111us) write_op.cc:270] Start()
1214 20:52:29.316225 (+ 16us) write_op.cc:276] Timestamp: P: 1765745540037905 usec, L: 0
1214 20:52:29.316353 (+ 128us) write_op.cc:270] Start()
1214 20:52:29.316369 (+ 16us) write_op.cc:276] Timestamp: P: 1765745540057942 usec, L: 0
1214 20:52:29.316478 (+ 109us) write_op.cc:270] Start()
1214 20:52:29.316502 (+ 24us) write_op.cc:276] Timestamp: P: 1765745540059171 usec, L: 0
1214 20:52:29.316624 (+ 122us) write_op.cc:270] Start()
1214 20:52:29.316642 (+ 18us) write_op.cc:276] Timestamp: P: 1765745540076765 usec, L: 0
1214 20:52:29.316766 (+ 124us) write_op.cc:270] Start()
1214 20:52:29.316783 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540081713 usec, L: 0
1214 20:52:29.316896 (+ 113us) write_op.cc:270] Start()
1214 20:52:29.316914 (+ 18us) write_op.cc:276] Timestamp: P: 1765745540088456 usec, L: 0
1214 20:52:29.317027 (+ 113us) write_op.cc:270] Start()
1214 20:52:29.317044 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540112944 usec, L: 0
1214 20:52:29.328077 (+ 11033us) write_op.cc:270] Start()
1214 20:52:29.328134 (+ 57us) write_op.cc:276] Timestamp: P: 1765745540114079 usec, L: 0
1214 20:52:29.328394 (+ 260us) write_op.cc:270] Start()
1214 20:52:29.328414 (+ 20us) write_op.cc:276] Timestamp: P: 1765745540165272 usec, L: 0
1214 20:52:29.328545 (+ 131us) write_op.cc:270] Start()
1214 20:52:29.328561 (+ 16us) write_op.cc:276] Timestamp: P: 1765745540175550 usec, L: 0
1214 20:52:29.328682 (+ 121us) write_op.cc:270] Start()
1214 20:52:29.328701 (+ 19us) write_op.cc:276] Timestamp: P: 1765745540176211 usec, L: 0
1214 20:52:29.328824 (+ 123us) write_op.cc:270] Start()
1214 20:52:29.328841 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540274749 usec, L: 0
1214 20:52:29.328961 (+ 120us) write_op.cc:270] Start()
1214 20:52:29.328977 (+ 16us) write_op.cc:276] Timestamp: P: 1765745540279839 usec, L: 0
1214 20:52:29.329077 (+ 100us) write_op.cc:270] Start()
1214 20:52:29.329093 (+ 16us) write_op.cc:276] Timestamp: P: 1765745540353394 usec, L: 0
1214 20:52:29.329229 (+ 136us) write_op.cc:270] Start()
1214 20:52:29.329252 (+ 23us) write_op.cc:276] Timestamp: P: 1765745540354313 usec, L: 0
1214 20:52:29.329368 (+ 116us) write_op.cc:270] Start()
1214 20:52:29.329384 (+ 16us) write_op.cc:276] Timestamp: P: 1765745540354960 usec, L: 0
1214 20:52:29.329483 (+ 99us) write_op.cc:270] Start()
1214 20:52:29.329498 (+ 15us) write_op.cc:276] Timestamp: P: 1765745540400138 usec, L: 0
1214 20:52:29.329600 (+ 102us) write_op.cc:270] Start()
1214 20:52:29.329642 (+ 42us) write_op.cc:276] Timestamp: P: 1765745540477954 usec, L: 0
1214 20:52:29.329777 (+ 135us) write_op.cc:270] Start()
1214 20:52:29.329793 (+ 16us) write_op.cc:276] Timestamp: P: 1765745540512084 usec, L: 0
1214 20:52:29.329928 (+ 135us) write_op.cc:270] Start()
1214 20:52:29.329946 (+ 18us) write_op.cc:276] Timestamp: P: 1765745540525950 usec, L: 0
1214 20:52:29.330076 (+ 130us) write_op.cc:270] Start()
1214 20:52:29.330093 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540546608 usec, L: 0
1214 20:52:29.330216 (+ 123us) write_op.cc:270] Start()
1214 20:52:29.330233 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540665774 usec, L: 0
1214 20:52:29.330341 (+ 108us) write_op.cc:270] Start()
1214 20:52:29.330357 (+ 16us) write_op.cc:276] Timestamp: P: 1765745540682485 usec, L: 0
1214 20:52:29.330501 (+ 144us) write_op.cc:270] Start()
1214 20:52:29.330519 (+ 18us) write_op.cc:276] Timestamp: P: 1765745540683153 usec, L: 0
1214 20:52:29.330647 (+ 128us) write_op.cc:270] Start()
1214 20:52:29.330665 (+ 18us) write_op.cc:276] Timestamp: P: 1765745540683763 usec, L: 0
1214 20:52:29.330785 (+ 120us) write_op.cc:270] Start()
1214 20:52:29.330802 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540684317 usec, L: 0
1214 20:52:29.330917 (+ 115us) write_op.cc:270] Start()
1214 20:52:29.330937 (+ 20us) write_op.cc:276] Timestamp: P: 1765745540684856 usec, L: 0
1214 20:52:29.331043 (+ 106us) write_op.cc:270] Start()
1214 20:52:29.331060 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540693429 usec, L: 0
1214 20:52:29.331192 (+ 132us) write_op.cc:270] Start()
1214 20:52:29.331210 (+ 18us) write_op.cc:276] Timestamp: P: 1765745540701253 usec, L: 0
1214 20:52:29.331324 (+ 114us) write_op.cc:270] Start()
1214 20:52:29.331342 (+ 18us) write_op.cc:276] Timestamp: P: 1765745540742587 usec, L: 0
1214 20:52:29.331453 (+ 111us) write_op.cc:270] Start()
1214 20:52:29.331470 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540743229 usec, L: 0
1214 20:52:29.331596 (+ 126us) write_op.cc:270] Start()
1214 20:52:29.331616 (+ 20us) write_op.cc:276] Timestamp: P: 1765745540750341 usec, L: 0
1214 20:52:29.331747 (+ 131us) write_op.cc:270] Start()
1214 20:52:29.331767 (+ 20us) write_op.cc:276] Timestamp: P: 1765745540751013 usec, L: 0
1214 20:52:29.331917 (+ 150us) write_op.cc:270] Start()
1214 20:52:29.331937 (+ 20us) write_op.cc:276] Timestamp: P: 1765745540751571 usec, L: 0
1214 20:52:29.332103 (+ 166us) write_op.cc:270] Start()
1214 20:52:29.332123 (+ 20us) write_op.cc:276] Timestamp: P: 1765745540792444 usec, L: 0
1214 20:52:29.332242 (+ 119us) write_op.cc:270] Start()
1214 20:52:29.332260 (+ 18us) write_op.cc:276] Timestamp: P: 1765745540793233 usec, L: 0
1214 20:52:29.332386 (+ 126us) write_op.cc:270] Start()
1214 20:52:29.332404 (+ 18us) write_op.cc:276] Timestamp: P: 1765745540793761 usec, L: 0
1214 20:52:29.332528 (+ 124us) write_op.cc:270] Start()
1214 20:52:29.332552 (+ 24us) write_op.cc:276] Timestamp: P: 1765745540804908 usec, L: 0
1214 20:52:29.338675 (+ 6123us) write_op.cc:270] Start()
1214 20:52:29.338809 (+ 134us) write_op.cc:276] Timestamp: P: 1765745540805531 usec, L: 0
1214 20:52:29.340824 (+ 2015us) write_op.cc:270] Start()
1214 20:52:29.340852 (+ 28us) write_op.cc:276] Timestamp: P: 1765745540806203 usec, L: 0
1214 20:52:29.342868 (+ 2016us) write_op.cc:270] Start()
1214 20:52:29.342898 (+ 30us) write_op.cc:276] Timestamp: P: 1765745540806848 usec, L: 0
1214 20:52:29.343058 (+ 160us) write_op.cc:270] Start()
1214 20:52:29.343079 (+ 21us) write_op.cc:276] Timestamp: P: 1765745540868310 usec, L: 0
1214 20:52:29.344053 (+ 974us) write_op.cc:270] Start()
1214 20:52:29.344078 (+ 25us) write_op.cc:276] Timestamp: P: 1765745540871211 usec, L: 0
1214 20:52:29.344201 (+ 123us) write_op.cc:270] Start()
1214 20:52:29.344218 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540882224 usec, L: 0
1214 20:52:29.344331 (+ 113us) write_op.cc:270] Start()
1214 20:52:29.344348 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540882890 usec, L: 0
1214 20:52:29.344453 (+ 105us) write_op.cc:270] Start()
1214 20:52:29.344470 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540883536 usec, L: 0
1214 20:52:29.344585 (+ 115us) write_op.cc:270] Start()
1214 20:52:29.344602 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540997234 usec, L: 0
1214 20:52:29.344727 (+ 125us) write_op.cc:270] Start()
1214 20:52:29.344744 (+ 17us) write_op.cc:276] Timestamp: P: 1765745540998259 usec, L: 0
1214 20:52:29.344845 (+ 101us) write_op.cc:270] Start()
1214 20:52:29.344871 (+ 26us) write_op.cc:276] Timestamp: P: 1765745540998808 usec, L: 0
1214 20:52:29.344991 (+ 120us) write_op.cc:270] Start()
1214 20:52:29.345007 (+ 16us) write_op.cc:276] Timestamp: P: 1765745541009325 usec, L: 0
1214 20:52:29.345134 (+ 127us) write_op.cc:270] Start()
1214 20:52:29.345151 (+ 17us) write_op.cc:276] Timestamp: P: 1765745541058511 usec, L: 0
1214 20:52:29.345259 (+ 108us) write_op.cc:270] Start()
1214 20:52:29.345274 (+ 15us) write_op.cc:276] Timestamp: P: 1765745541075663 usec, L: 0
1214 20:52:29.345389 (+ 115us) write_op.cc:270] Start()
1214 20:52:29.345405 (+ 16us) write_op.cc:276] Timestamp: P: 1765745541112770 usec, L: 0
1214 20:52:29.345500 (+ 95us) write_op.cc:270] Start()
1214 20:52:29.345515 (+ 15us) write_op.cc:276] Timestamp: P: 1765745541131012 usec, L: 0
1214 20:52:29.345662 (+ 147us) write_op.cc:270] Start()
1214 20:52:29.345680 (+ 18us) write_op.cc:276] Timestamp: P: 1765745541206794 usec, L: 0
1214 20:52:29.345813 (+ 133us) write_op.cc:270] Start()
1214 20:52:29.345830 (+ 17us) write_op.cc:276] Timestamp: P: 1765745541210655 usec, L: 0
1214 20:52:29.345951 (+ 121us) write_op.cc:270] Start()
1214 20:52:29.345968 (+ 17us) write_op.cc:276] Timestamp: P: 1765745541231790 usec, L: 0
1214 20:52:29.346079 (+ 111us) write_op.cc:270] Start()
1214 20:52:29.346096 (+ 17us) write_op.cc:276] Timestamp: P: 1765745541391567 usec, L: 0
1214 20:52:29.346198 (+ 102us) write_op.cc:270] Start()
1214 20:52:29.346213 (+ 15us) write_op.cc:276] Timestamp: P: 1765745541445659 usec, L: 0
1214 20:52:29.346310 (+ 97us) write_op.cc:270] Start()
1214 20:52:29.346325 (+ 15us) write_op.cc:276] Timestamp: P: 1765745541674206 usec, L: 0
1214 20:52:29.346425 (+ 100us) write_op.cc:270] Start()
1214 20:52:29.346441 (+ 16us) write_op.cc:276] Timestamp: P: 1765745541879611 usec, L: 0
1214 20:52:29.346541 (+ 100us) write_op.cc:270] Start()
1214 20:52:29.346557 (+ 16us) write_op.cc:276] Timestamp: P: 1765745542002322 usec, L: 0
1214 20:52:29.346731 (+ 174us) write_op.cc:270] Start()
1214 20:52:29.346752 (+ 21us) write_op.cc:276] Timestamp: P: 1765745542003027 usec, L: 0
1214 20:52:29.346891 (+ 139us) write_op.cc:270] Start()
1214 20:52:29.346911 (+ 20us) write_op.cc:276] Timestamp: P: 1765745542041631 usec, L: 0
1214 20:52:29.347036 (+ 125us) write_op.cc:270] Start()
1214 20:52:29.347053 (+ 17us) write_op.cc:276] Timestamp: P: 1765745542113112 usec, L: 0
1214 20:52:29.364569 (+ 17516us) write_op.cc:270] Start()
1214 20:52:29.364619 (+ 50us) write_op.cc:276] Timestamp: P: 1765745542115733 usec, L: 0
1214 20:52:29.364856 (+ 237us) write_op.cc:270] Start()
1214 20:52:29.364893 (+ 37us) write_op.cc:276] Timestamp: P: 1765745542120197 usec, L: 0
1214 20:52:29.365041 (+ 148us) write_op.cc:270] Start()
1214 20:52:29.365058 (+ 17us) write_op.cc:276] Timestamp: P: 1765745542126714 usec, L: 0
1214 20:52:29.365191 (+ 133us) write_op.cc:270] Start()
1214 20:52:29.365207 (+ 16us) write_op.cc:276] Timestamp: P: 1765745542134742 usec, L: 0
1214 20:52:29.365322 (+ 115us) write_op.cc:270] Start()
1214 20:52:29.365358 (+ 36us) write_op.cc:276] Timestamp: P: 1765745542135855 usec, L: 0
1214 20:52:29.365469 (+ 111us) write_op.cc:270] Start()
1214 20:52:29.365485 (+ 16us) write_op.cc:276] Timestamp: P: 1765745542208199 usec, L: 0
1214 20:52:29.365615 (+ 130us) write_op.cc:270] Start()
1214 20:52:29.365641 (+ 26us) write_op.cc:276] Timestamp: P: 1765745542276512 usec, L: 0
1214 20:52:29.365760 (+ 119us) write_op.cc:270] Start()
1214 20:52:29.365778 (+ 18us) write_op.cc:276] Timestamp: P: 1765745542307436 usec, L: 0
1214 20:52:29.365930 (+ 152us) write_op.cc:270] Start()
1214 20:52:29.365950 (+ 20us) write_op.cc:276] Timestamp: P: 1765745542309102 usec, L: 0
1214 20:52:29.366115 (+ 165us) write_op.cc:270] Start()
1214 20:52:29.366135 (+ 20us) write_op.cc:276] Timestamp: P: 1765745542340986 usec, L: 0
1214 20:52:29.366268 (+ 133us) write_op.cc:270] Start()
1214 20:52:29.366284 (+ 16us) write_op.cc:276] Timestamp: P: 1765745542345026 usec, L: 0
1214 20:52:29.366388 (+ 104us) write_op.cc:270] Start()
1214 20:52:29.366403 (+ 15us) write_op.cc:276] Timestamp: P: 1765745542359659 usec, L: 0
1214 20:52:29.366503 (+ 100us) write_op.cc:270] Start()
1214 20:52:29.366518 (+ 15us) write_op.cc:276] Timestamp: P: 1765745542380006 usec, L: 0
1214 20:52:29.366626 (+ 108us) write_op.cc:270] Start()
1214 20:52:29.366643 (+ 17us) write_op.cc:276] Timestamp: P: 1765745542402239 usec, L: 0
1214 20:52:29.366760 (+ 117us) write_op.cc:270] Start()
1214 20:52:29.366782 (+ 22us) write_op.cc:276] Timestamp: P: 1765745542418643 usec, L: 0
1214 20:52:29.366890 (+ 108us) write_op.cc:270] Start()
1214 20:52:29.366906 (+ 16us) write_op.cc:276] Timestamp: P: 1765745542525648 usec, L: 0
1214 20:52:29.367013 (+ 107us) write_op.cc:270] Start()
1214 20:52:29.367029 (+ 16us) write_op.cc:276] Timestamp: P: 1765745542538056 usec, L: 0
1214 20:52:29.374444 (+ 7415us) write_op.cc:270] Start()
1214 20:52:29.374495 (+ 51us) write_op.cc:276] Timestamp: P: 1765745542541003 usec, L: 0
1214 20:52:29.374728 (+ 233us) write_op.cc:270] Start()
1214 20:52:29.374754 (+ 26us) write_op.cc:276] Timestamp: P: 1765745542556675 usec, L: 0
1214 20:52:29.379531 (+ 4777us) write_op.cc:270] Start()
1214 20:52:29.379568 (+ 37us) write_op.cc:276] Timestamp: P: 1765745542562584 usec, L: 0
1214 20:52:29.379764 (+ 196us) write_op.cc:270] Start()
1214 20:52:29.379787 (+ 23us) write_op.cc:276] Timestamp: P: 1765745542575994 usec, L: 0
1214 20:52:29.379930 (+ 143us) write_op.cc:270] Start()
1214 20:52:29.379951 (+ 21us) write_op.cc:276] Timestamp: P: 1765745542577573 usec, L: 0
1214 20:52:29.380076 (+ 125us) write_op.cc:270] Start()
1214 20:52:29.380132 (+ 56us) write_op.cc:276] Timestamp: P: 1765745542639578 usec, L: 0
1214 20:52:29.380269 (+ 137us) write_op.cc:270] Start()
1214 20:52:29.380290 (+ 21us) write_op.cc:276] Timestamp: P: 1765745542667890 usec, L: 0
1214 20:52:29.390501 (+ 10211us) write_op.cc:270] Start()
1214 20:52:29.390548 (+ 47us) write_op.cc:276] Timestamp: P: 1765745542668581 usec, L: 0
1214 20:52:29.391531 (+ 983us) write_op.cc:270] Start()
1214 20:52:29.391569 (+ 38us) write_op.cc:276] Timestamp: P: 1765745542669138 usec, L: 0
1214 20:52:29.391766 (+ 197us) write_op.cc:270] Start()
1214 20:52:29.391788 (+ 22us) write_op.cc:276] Timestamp: P: 1765745542669759 usec, L: 0
1214 20:52:29.391945 (+ 157us) write_op.cc:270] Start()
1214 20:52:29.391963 (+ 18us) write_op.cc:276] Timestamp: P: 1765745542715925 usec, L: 0
1214 20:52:29.408790 (+ 16827us) write_op.cc:270] Start()
1214 20:52:29.409066 (+ 276us) write_op.cc:276] Timestamp: P: 1765745542716700 usec, L: 0
1214 20:52:29.424378 (+ 15312us) write_op.cc:270] Start()
1214 20:52:29.424420 (+ 42us) write_op.cc:276] Timestamp: P: 1765745542717255 usec, L: 0
1214 20:52:29.425327 (+ 907us) write_op.cc:270] Start()
1214 20:52:29.425357 (+ 30us) write_op.cc:276] Timestamp: P: 1765745542745801 usec, L: 0
1214 20:52:29.425511 (+ 154us) write_op.cc:270] Start()
1214 20:52:29.425533 (+ 22us) write_op.cc:276] Timestamp: P: 1765745542832097 usec, L: 0
1214 20:52:29.425692 (+ 159us) write_op.cc:270] Start()
1214 20:52:29.425715 (+ 23us) write_op.cc:276] Timestamp: P: 1765745542842594 usec, L: 0
1214 20:52:29.425880 (+ 165us) write_op.cc:270] Start()
1214 20:52:29.425902 (+ 22us) write_op.cc:276] Timestamp: P: 1765745542848559 usec, L: 0
1214 20:52:29.426034 (+ 132us) write_op.cc:270] Start()
1214 20:52:29.426054 (+ 20us) write_op.cc:276] Timestamp: P: 1765745542851781 usec, L: 0
1214 20:52:29.427298 (+ 1244us) write_op.cc:270] Start()
1214 20:52:29.427322 (+ 24us) write_op.cc:276] Timestamp: P: 1765745542856353 usec, L: 0
1214 20:52:29.427443 (+ 121us) write_op.cc:270] Start()
1214 20:52:29.427461 (+ 18us) write_op.cc:276] Timestamp: P: 1765745542913139 usec, L: 0
1214 20:52:29.427593 (+ 132us) write_op.cc:270] Start()
1214 20:52:29.427612 (+ 19us) write_op.cc:276] Timestamp: P: 1765745542915408 usec, L: 0
1214 20:52:29.427729 (+ 117us) write_op.cc:270] Start()
1214 20:52:29.427750 (+ 21us) write_op.cc:276] Timestamp: P: 1765745542953763 usec, L: 0
1214 20:52:29.427888 (+ 138us) write_op.cc:270] Start()
1214 20:52:29.427908 (+ 20us) write_op.cc:276] Timestamp: P: 1765745543036065 usec, L: 0
1214 20:52:29.428070 (+ 162us) write_op.cc:270] Start()
1214 20:52:29.428271 (+ 201us) write_op.cc:276] Timestamp: P: 1765745543050092 usec, L: 0
1214 20:52:29.428529 (+ 258us) write_op.cc:270] Start()
1214 20:52:29.428555 (+ 26us) write_op.cc:276] Timestamp: P: 1765745543060469 usec, L: 0
1214 20:52:29.430493 (+ 1938us) write_op.cc:270] Start()
1214 20:52:29.430531 (+ 38us) write_op.cc:276] Timestamp: P: 1765745543095205 usec, L: 0
1214 20:52:29.432089 (+ 1558us) write_op.cc:270] Start()
1214 20:52:29.432118 (+ 29us) write_op.cc:276] Timestamp: P: 1765745543139915 usec, L: 0
1214 20:52:29.432260 (+ 142us) write_op.cc:270] Start()
1214 20:52:29.432280 (+ 20us) write_op.cc:276] Timestamp: P: 1765745543141397 usec, L: 0
1214 20:52:29.432417 (+ 137us) write_op.cc:270] Start()
1214 20:52:29.432439 (+ 22us) write_op.cc:276] Timestamp: P: 1765745543244811 usec, L: 0
1214 20:52:29.433688 (+ 1249us) write_op.cc:270] Start()
1214 20:52:29.433719 (+ 31us) write_op.cc:276] Timestamp: P: 1765745543246013 usec, L: 0
1214 20:52:29.433887 (+ 168us) write_op.cc:270] Start()
1214 20:52:29.433908 (+ 21us) write_op.cc:276] Timestamp: P: 1765745543248074 usec, L: 0
1214 20:52:29.434040 (+ 132us) write_op.cc:270] Start()
1214 20:52:29.434061 (+ 21us) write_op.cc:276] Timestamp: P: 1765745543333355 usec, L: 0
1214 20:52:29.435020 (+ 959us) write_op.cc:270] Start()
1214 20:52:29.435053 (+ 33us) write_op.cc:276] Timestamp: P: 1765745543334107 usec, L: 0
1214 20:52:29.435317 (+ 264us) write_op.cc:270] Start()
1214 20:52:29.435348 (+ 31us) write_op.cc:276] Timestamp: P: 1765745543403235 usec, L: 0
1214 20:52:29.435579 (+ 231us) write_op.cc:270] Start()
1214 20:52:29.435600 (+ 21us) write_op.cc:276] Timestamp: P: 1765745543406084 usec, L: 0
1214 20:52:29.435726 (+ 126us) write_op.cc:270] Start()
1214 20:52:29.435747 (+ 21us) write_op.cc:276] Timestamp: P: 1765745543407456 usec, L: 0
1214 20:52:29.435925 (+ 178us) write_op.cc:270] Start()
1214 20:52:29.435949 (+ 24us) write_op.cc:276] Timestamp: P: 1765745543408368 usec, L: 0
1214 20:52:29.438191 (+ 2242us) write_op.cc:270] Start()
1214 20:52:29.438224 (+ 33us) write_op.cc:276] Timestamp: P: 1765745543431807 usec, L: 0
1214 20:52:29.438418 (+ 194us) write_op.cc:270] Start()
1214 20:52:29.438443 (+ 25us) write_op.cc:276] Timestamp: P: 1765745543438156 usec, L: 0
1214 20:52:29.438602 (+ 159us) write_op.cc:270] Start()
1214 20:52:29.438625 (+ 23us) write_op.cc:276] Timestamp: P: 1765745543479860 usec, L: 0
1214 20:52:29.439465 (+ 840us) write_op.cc:270] Start()
1214 20:52:29.439490 (+ 25us) write_op.cc:276] Timestamp: P: 1765745543483143 usec, L: 0
1214 20:52:29.439627 (+ 137us) write_op.cc:270] Start()
1214 20:52:29.439648 (+ 21us) write_op.cc:276] Timestamp: P: 1765745543484382 usec, L: 0
1214 20:52:29.439789 (+ 141us) write_op.cc:270] Start()
1214 20:52:29.439811 (+ 22us) write_op.cc:276] Timestamp: P: 1765745543485376 usec, L: 0
1214 20:52:29.439961 (+ 150us) write_op.cc:270] Start()
1214 20:52:29.439981 (+ 20us) write_op.cc:276] Timestamp: P: 1765745543521884 usec, L: 0
1214 20:52:29.440138 (+ 157us) write_op.cc:270] Start()
1214 20:52:29.440158 (+ 20us) write_op.cc:276] Timestamp: P: 1765745543532191 usec, L: 0
1214 20:52:29.440283 (+ 125us) write_op.cc:270] Start()
1214 20:52:29.440302 (+ 19us) write_op.cc:276] Timestamp: P: 1765745543533459 usec, L: 0
1214 20:52:29.440429 (+ 127us) write_op.cc:270] Start()
1214 20:52:29.440449 (+ 20us) write_op.cc:276] Timestamp: P: 1765745543534760 usec, L: 0
1214 20:52:29.440625 (+ 176us) write_op.cc:270] Start()
1214 20:52:29.440647 (+ 22us) write_op.cc:276] Timestamp: P: 1765745543558691 usec, L: 0
1214 20:52:29.440799 (+ 152us) write_op.cc:270] Start()
1214 20:52:29.440820 (+ 21us) write_op.cc:276] Timestamp: P: 1765745543587838 usec, L: 0
1214 20:52:29.440964 (+ 144us) write_op.cc:270] Start()
1214 20:52:29.440984 (+ 20us) write_op.cc:276] Timestamp: P: 1765745543627492 usec, L: 0
1214 20:52:29.441121 (+ 137us) write_op.cc:270] Start()
1214 20:52:29.441141 (+ 20us) write_op.cc:276] Timestamp: P: 1765745543649128 usec, L: 0
1214 20:52:29.441267 (+ 126us) write_op.cc:270] Start()
1214 20:52:29.441286 (+ 19us) write_op.cc:276] Timestamp: P: 1765745543655136 usec, L: 0
1214 20:52:29.441419 (+ 133us) write_op.cc:270] Start()
1214 20:52:29.441438 (+ 19us) write_op.cc:276] Timestamp: P: 1765745543677308 usec, L: 0
1214 20:52:29.441562 (+ 124us) write_op.cc:270] Start()
1214 20:52:29.441581 (+ 19us) write_op.cc:276] Timestamp: P: 1765745
I20251214 20:52:30.939944 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_c50d34acaf2e4ec8be9f4c60d90ec25c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.944011 449 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_4f6c749899d64d8884fd2ac83cfb13fd" } modify_external_catalogs: true
I20251214 20:52:30.944296 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_81e31e2b08a1484c8adb4f139e0edfd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.948153 1203 tablet_bootstrap.cc:492] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:30.948369 1203 ts_tablet_manager.cc:1403] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.086s user 0.010s sys 0.002s
I20251214 20:52:30.949148 1203 raft_consensus.cc:359] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:30.949446 1203 raft_consensus.cc:385] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:30.949802 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_a1cc2f56735544ad8ae3425ad4d00fdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.943567 450 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90] alter complete (version 1)
I20251214 20:52:30.950651 1203 raft_consensus.cc:740] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:30.951093 449 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_4f6c749899d64d8884fd2ac83cfb13fd" } modify_external_catalogs: true
I20251214 20:52:30.951422 1203 consensus_queue.cc:260] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:30.951750 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81e31e2b08a1484c8adb4f139e0edfd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:30.953137 454 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1cc2f56735544ad8ae3425ad4d00fdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.956090 1203 ts_tablet_manager.cc:1434] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20251214 20:52:30.959205 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c50d34acaf2e4ec8be9f4c60d90ec25c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.959306 1203 tablet_bootstrap.cc:492] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:30.960811 1203 tablet_bootstrap.cc:654] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:30.968248 450 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_da9e4432a9254052ace01a09e425f97b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:30.973170 1856 ts_tablet_manager.cc:1929] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:30.973311 1856 log.cc:1199] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/9e1835374cfa48939d72476870accc3b
I20251214 20:52:30.973614 1856 ts_tablet_manager.cc:1950] T 9e1835374cfa48939d72476870accc3b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:30.975684 450 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da9e4432a9254052ace01a09e425f97b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.977380 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 532) took 1143 ms. Trace:
I20251214 20:52:30.977471 1856 rpcz_store.cc:276] 1214 20:52:29.834115 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.834161 (+ 46us) service_pool.cc:225] Handling call
1214 20:52:30.693402 (+859241us) ts_tablet_manager.cc:1151] Deleting tablet 9e1835374cfa48939d72476870accc3b
1214 20:52:30.693413 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:30.766754 (+ 73341us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.972988 (+206234us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:30.977360 (+ 4372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":837751}
I20251214 20:52:30.977856 1856 tablet_replica.cc:333] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:30.978061 1856 raft_consensus.cc:2243] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251214 20:52:30.978037 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:35099, user_credentials={real_user=slave}} blocked reactor thread for 84491.7us
I20251214 20:52:30.978195 1856 raft_consensus.cc:2272] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:30.978677 1856 ts_tablet_manager.cc:1916] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:30.979488 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 9e1835374cfa48939d72476870accc3b (table default.Table_1ba7876ea93447038c0de7f37fb52825 [id=ba545f2c5ba949709f18c486d993faab]) successfully deleted
I20251214 20:52:30.988059 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_5b82f109385a42b0a04a5272bc5f0156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:30.988890 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b82f109385a42b0a04a5272bc5f0156 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:30.993700 1203 tablet_bootstrap.cc:492] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:30.995102 1203 ts_tablet_manager.cc:1403] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.002s
I20251214 20:52:30.995997 1203 raft_consensus.cc:359] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.001137 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 453cf734f8b34b59aa246d618ffeade5 (table default.Table_93fe3196f8c94960a5c5ee9e1f3923f1 [id=611e3a3e47ba4f91a705e9b8d3c323f3]) successfully deleted
I20251214 20:52:30.878821 1202 consensus_queue.cc:260] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.003396 888 tablet_service.cc:1505] Processing CreateTablet for tablet c75e6701210e4d688a96678bf042860a (DEFAULT_TABLE table=default.Table_040e46c43e8446f48a60e34f0892bf57 [id=2b6d1f218393414d868d37d390b35b58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.003516 1977 raft_consensus.cc:2243] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:31.003734 1977 raft_consensus.cc:2272] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:31.003883 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75e6701210e4d688a96678bf042860a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.007398 2779 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:31.007629 2779 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:31.009887 878 tablet_service.cc:1505] Processing CreateTablet for tablet 91798ce3e88e4f82b59c52222dd83aef (DEFAULT_TABLE table=default.Table_8cd3b56154ec4d9fac4ed571a2287cfa [id=39df4da3b39545d3ba2c9345618ae532]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.010311 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91798ce3e88e4f82b59c52222dd83aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.012341 1203 raft_consensus.cc:385] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.012733 1203 raft_consensus.cc:740] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.014552 1203 consensus_queue.cc:260] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.015910 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80e28c4d87774256acb8e9638ded8689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.024323 1203 ts_tablet_manager.cc:1434] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.029s user 0.000s sys 0.001s
W20251214 20:52:31.031327 452 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2398f176ee044c15841eaa40cb74974c consensus state has no leader
I20251214 20:52:31.032915 1203 tablet_bootstrap.cc:492] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.034337 1203 tablet_bootstrap.cc:654] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:31.034837 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba322cb6ee084bc39788ddd439f5f15d consensus state has no leader
I20251214 20:52:31.038856 879 tablet_service.cc:1505] Processing CreateTablet for tablet 243528f1aeac425c8be47f9fae802f52 (DEFAULT_TABLE table=default.Table_8923490b824a47139d8f739d65e4845b [id=2ffffc8a53e04583a2f93b287fe083bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.040813 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243528f1aeac425c8be47f9fae802f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.040371 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f75657d7bc44f6393c55ab119eb7dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:31.040035 881 tablet_service.cc:1505] Processing CreateTablet for tablet 74d2f8e5481d40b39cc4df34b0d4f819 (DEFAULT_TABLE table=default.Table_0fb8f39742d245ac9d20fc7d6bf7e7c3 [id=552c8b213ba64d4bbdd3363cdc6625ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.042807 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d2f8e5481d40b39cc4df34b0d4f819. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:30.785099 748 tablet_service.cc:1552] Processing DeleteTablet for tablet e90d11d69b8d4fd49150aa67a8190e5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:31.048246 1977 ts_tablet_manager.cc:1916] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:31.055831 748 tablet_service.cc:1505] Processing CreateTablet for tablet 74d2f8e5481d40b39cc4df34b0d4f819 (DEFAULT_TABLE table=default.Table_0fb8f39742d245ac9d20fc7d6bf7e7c3 [id=552c8b213ba64d4bbdd3363cdc6625ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.058881 1203 tablet_bootstrap.cc:492] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.059181 1203 ts_tablet_manager.cc:1403] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.002s
I20251214 20:52:31.059923 1203 raft_consensus.cc:359] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.060281 1203 raft_consensus.cc:385] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.060389 1203 raft_consensus.cc:740] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.060710 1203 consensus_queue.cc:260] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.061412 1203 ts_tablet_manager.cc:1434] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251214 20:52:31.065367 451 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67104e34df294ef5986171eca79cdc72 consensus state has no leader
I20251214 20:52:31.071348 453 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ccc4e873bf4247069a3b9bc25d2a0ca6" } modify_external_catalogs: true
I20251214 20:52:31.071687 453 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ccc4e873bf4247069a3b9bc25d2a0ca6" } modify_external_catalogs: true
I20251214 20:52:31.073004 455 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_02304abff69e46f68c7152bfc33cb44d" } new_table_name: "default.Table_cb922bf98617480aa9363c3f08d918f3" modify_external_catalogs: true
I20251214 20:52:31.074870 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d2f8e5481d40b39cc4df34b0d4f819. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.083175 448 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_87794c3eaa7c4545acddac9ad4b4bc93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:31.084013 448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87794c3eaa7c4545acddac9ad4b4bc93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:31.085878 451 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d5446dd7bdf14b76ab6136efd369dd12" } modify_external_catalogs: true
I20251214 20:52:31.086054 451 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_d5446dd7bdf14b76ab6136efd369dd12" } modify_external_catalogs: true
I20251214 20:52:31.090886 447 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_78f00e8dd81a485cae8269bac0d08866"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:31.091612 447 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78f00e8dd81a485cae8269bac0d08866 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:31.101651 446 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_ac4df504ee284227bfd4ba2a23eaff65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:31.102691 446 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac4df504ee284227bfd4ba2a23eaff65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:31.106449 452 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d424e3387698434f92e84eb1d43f5949 (table default.Table_9098a11909c844dca6d4e80a284aa2e8 [id=1597e24e3fff4007ab66b44505c960d8]). Expected version 1 got 0
I20251214 20:52:31.108945 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 234a3f2c04e94149a1aeca848e6a4ce4 (DEFAULT_TABLE table=default.Table_bdfed01cc4cd4458be399de18a738f45 [id=164c3a6291b644789ae5905faca45b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.109445 1017 tablet_service.cc:1505] Processing CreateTablet for tablet ddd625b1cf4a4af191ad4e9a85f7b976 (DEFAULT_TABLE table=default.Table_58bb2f3098084d73bafbd51cde268a21 [id=1e84b30c44fc49ba8a8b35c7e3252761]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.111192 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 1dc5e36369574906a0602145b37aa8d6 (DEFAULT_TABLE table=default.Table_da61d1abe3f24498977b6cc43d2ed4df [id=800f51296fe14db2985b673976f6ec6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.117033 450 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_7e0acfc6d6e84dad84938bb81b1f375d" } new_table_name: "default.Table_c3aa8e3c31cb41dc8281962e7b5c46e1" modify_external_catalogs: true
I20251214 20:52:31.118747 454 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_309db467eb00418598443e6f7a007e44" } modify_external_catalogs: true
I20251214 20:52:31.119100 454 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_309db467eb00418598443e6f7a007e44" } modify_external_catalogs: true
I20251214 20:52:31.127456 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 6f231db5cc4a4060a996465defb23c6c (DEFAULT_TABLE table=default.Table_e277c40da9e24b7fabb8edd25ec3413d [id=e973536efad243e5b94f5994d57325b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.144999 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f75657d7bc44f6393c55ab119eb7dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:31.145210 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 348d8dcef97042279bc0d5ff31d3c137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:31.145323 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 1def5f0de5ce4eb0bed36f626c4f0683 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:31.145649 742 tablet_service.cc:1505] Processing CreateTablet for tablet 234a3f2c04e94149a1aeca848e6a4ce4 (DEFAULT_TABLE table=default.Table_bdfed01cc4cd4458be399de18a738f45 [id=164c3a6291b644789ae5905faca45b10]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:31.149053 1206 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.000s sys 0.017s
W20251214 20:52:31.149228 1206 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.763s user 0.000s sys 0.017s
W20251214 20:52:31.151996 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 343478us
I20251214 20:52:31.153945 1206 tablet_bootstrap.cc:492] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:31.154171 1206 ts_tablet_manager.cc:1403] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.804s user 0.001s sys 0.019s
I20251214 20:52:31.162099 2735 raft_consensus.cc:493] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.162307 2735 raft_consensus.cc:515] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.163087 2735 leader_election.cc:290] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.164542 2735 raft_consensus.cc:493] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.164749 2735 raft_consensus.cc:515] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.165323 2735 leader_election.cc:290] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.166615 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b47ca1a03944cf1847d3578856dafa7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:31.167315 989 leader_election.cc:343] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.175895 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6bdfbe4d15345de8be5de00bfe4ca07" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:31.176152 1050 raft_consensus.cc:3060] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:31.179939 2703 raft_consensus.cc:493] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.180114 2703 raft_consensus.cc:515] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.180768 2703 leader_election.cc:290] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.186887 1050 raft_consensus.cc:2468] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:31.188194 1203 tablet_bootstrap.cc:492] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.189994 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234a3f2c04e94149a1aeca848e6a4ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.190237 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f231db5cc4a4060a996465defb23c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.195264 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddd625b1cf4a4af191ad4e9a85f7b976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.196107 1203 tablet_bootstrap.cc:654] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.199252 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc5e36369574906a0602145b37aa8d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.203382 1856 ts_tablet_manager.cc:1929] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:31.203555 1856 log.cc:1199] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/492770a7b8b74ba49315f974cc54eedb
I20251214 20:52:31.204082 1856 ts_tablet_manager.cc:1950] T 492770a7b8b74ba49315f974cc54eedb P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:31.215342 1203 tablet_bootstrap.cc:492] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.220211 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc73d590531c409e9f60670435dc8223" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:31.220891 2330 ts_manager.cc:194] Registered new tserver with Master: 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:31.224603 2664 mvcc.cc:204] Tried to move back new op lower bound from 7232493757175713792 to 7232493717732380672. Current Snapshot: MvccSnapshot[applied={T|T < 7232493757175713792 or (T in {7232493757175713792})}]
W20251214 20:52:31.222309 989 leader_election.cc:343] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.217696 1203 ts_tablet_manager.cc:1403] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.003s
I20251214 20:52:31.228518 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234a3f2c04e94149a1aeca848e6a4ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.230121 1203 raft_consensus.cc:359] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.234333 1203 raft_consensus.cc:385] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.235002 1203 raft_consensus.cc:740] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.236706 1203 consensus_queue.cc:260] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.242026 1203 ts_tablet_manager.cc:1434] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251214 20:52:31.248080 1203 tablet_bootstrap.cc:492] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.254158 2735 raft_consensus.cc:493] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.254329 2735 raft_consensus.cc:515] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.255049 2735 leader_election.cc:290] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.255652 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274185a4115b458e8e9a376bdbcc870b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:31.256250 989 leader_election.cc:343] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.254658 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ebfc373fbf41b1920c271a15302484" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:31.261483 989 leader_election.cc:343] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.266860 1203 tablet_bootstrap.cc:654] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.280798 2774 raft_consensus.cc:493] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.280977 2774 raft_consensus.cc:515] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.281632 2774 leader_election.cc:290] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:31.288399 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 533) took 1436 ms. Trace:
I20251214 20:52:31.288766 1856 rpcz_store.cc:276] 1214 20:52:29.851919 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.851972 (+ 53us) service_pool.cc:225] Handling call
1214 20:52:30.977807 (+1125835us) ts_tablet_manager.cc:1151] Deleting tablet 492770a7b8b74ba49315f974cc54eedb
1214 20:52:30.977817 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:31.083118 (+105301us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:31.190109 (+106991us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:31.288358 (+ 98249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1122551}
I20251214 20:52:31.289427 1856 tablet_replica.cc:333] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:31.289718 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 492770a7b8b74ba49315f974cc54eedb (table default.Table_005b061b84dc4e7aa3fe295da6b99cd1 [id=d6c5c9fbf30b4477b502d001a4f3cd27]) successfully deleted
I20251214 20:52:31.289691 1856 raft_consensus.cc:2243] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:31.290155 1856 raft_consensus.cc:2272] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:31.290962 1856 ts_tablet_manager.cc:1916] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:31.292613 1203 tablet_bootstrap.cc:492] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.293730 1203 ts_tablet_manager.cc:1403] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.000s
I20251214 20:52:31.294554 1203 raft_consensus.cc:359] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.297406 1203 raft_consensus.cc:385] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.297541 1203 raft_consensus.cc:740] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.298781 1203 consensus_queue.cc:260] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.300758 1203 ts_tablet_manager.cc:1434] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251214 20:52:31.302136 1203 tablet_bootstrap.cc:492] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.311437 1203 tablet_bootstrap.cc:654] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:31.329368 440 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 566c6e4761d5486d964e276d151e32b0 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 566c6e4761d5486d964e276d151e32b0 consensus state has no leader
I20251214 20:52:31.330647 1203 tablet_bootstrap.cc:492] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.330816 1203 ts_tablet_manager.cc:1403] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
W20251214 20:52:31.330972 2548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.006s sys 0.020s
I20251214 20:52:31.331750 1203 raft_consensus.cc:359] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.331938 1203 raft_consensus.cc:385] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.332003 1203 raft_consensus.cc:740] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.332238 1203 consensus_queue.cc:260] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.332726 1203 ts_tablet_manager.cc:1434] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:31.333031 1203 tablet_bootstrap.cc:492] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.334717 1203 tablet_bootstrap.cc:654] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:31.331717 2548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s user 0.006s sys 0.020s
I20251214 20:52:31.354377 1203 tablet_bootstrap.cc:492] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.354599 1203 ts_tablet_manager.cc:1403] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20251214 20:52:31.355480 1203 raft_consensus.cc:359] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.355690 1203 raft_consensus.cc:385] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.355767 1203 raft_consensus.cc:740] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.356000 1203 consensus_queue.cc:260] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.356508 1203 ts_tablet_manager.cc:1434] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:31.356807 1203 tablet_bootstrap.cc:492] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.358484 1203 tablet_bootstrap.cc:654] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.388602 2735 raft_consensus.cc:493] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.388844 2735 raft_consensus.cc:515] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.389604 2735 leader_election.cc:290] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.389663 1977 ts_tablet_manager.cc:1929] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:31.389806 1977 log.cc:1199] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/d13f93bb5e5f4b59bcf9f4bfed070f25
I20251214 20:52:31.390177 1977 ts_tablet_manager.cc:1950] T d13f93bb5e5f4b59bcf9f4bfed070f25 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:31.399312 2735 raft_consensus.cc:493] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.399484 2735 raft_consensus.cc:515] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.399587 1203 tablet_bootstrap.cc:492] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.399749 1203 ts_tablet_manager.cc:1403] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.012s
I20251214 20:52:31.399711 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65e79db74fa046698507c83416ee1669" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:31.400173 2735 leader_election.cc:290] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:31.400331 989 leader_election.cc:343] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.400578 1203 raft_consensus.cc:359] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.400854 1203 raft_consensus.cc:385] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.400844 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b69bb4d5979f492683fc13238a668b9f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:31.400966 1203 raft_consensus.cc:740] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.401281 1203 consensus_queue.cc:260] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:31.401506 989 leader_election.cc:343] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.402806 1203 ts_tablet_manager.cc:1434] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251214 20:52:31.403784 1203 tablet_bootstrap.cc:492] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.406101 1203 tablet_bootstrap.cc:654] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.421166 737 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37360 (request call id 252) took 1428 ms. Trace:
I20251214 20:52:31.425202 1203 tablet_bootstrap.cc:492] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.425365 1203 ts_tablet_manager.cc:1403] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20251214 20:52:31.426093 1203 raft_consensus.cc:359] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.426263 1203 raft_consensus.cc:385] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.426313 1203 raft_consensus.cc:740] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.426514 1203 consensus_queue.cc:260] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.427008 1203 ts_tablet_manager.cc:1434] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:31.427340 1203 tablet_bootstrap.cc:492] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.429147 1203 tablet_bootstrap.cc:654] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.424407 737 rpcz_store.cc:276] 1214 20:52:29.992995 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.993047 (+ 52us) service_pool.cc:225] Handling call
1214 20:52:30.799754 (+806707us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x616000120a54. stack: 00007f95284c1ee6 00007f95284c1ca0 00007f952749d3ad 0000000000984f94 0000000000984f5d 000000000098492a 00007f953437a2c0 00007f953822ab2f 00007f953264fc36 00007f953264f965 00007f952f48056e 00007f952f47f44e 00007f952f485685 00007f952f485faf 00007f952f485dd8 0000000000a8970c
1214 20:52:31.421110 (+621356us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":758624640}
I20251214 20:52:31.448472 1202 ts_tablet_manager.cc:1434] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.572s user 0.015s sys 0.022s
W20251214 20:52:31.448649 1202 ts_tablet_manager.cc:1467] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: Tablet startup took 572ms
W20251214 20:52:31.448745 1202 ts_tablet_manager.cc:1469] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: Trace:
1214 20:52:15.665869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:15.665985 (+ 116us) service_pool.cc:225] Handling call
1214 20:52:15.666653 (+ 668us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1214 20:52:15.666684 (+ 31us) ts_tablet_manager.cc:682] Creating new metadata...
1214 20:52:15.670215 (+ 3531us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:15.716964 (+ 46749us) tablet_replica.cc:206] Creating consensus instance
1214 20:52:15.717117 (+ 153us) inbound_call.cc:173] Queueing success response
1214 20:52:30.649063 (+14931946us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1214 20:52:30.876064 (+227001us) ts_tablet_manager.cc:1435] Starting tablet replica
1214 20:52:30.876233 (+ 169us) tablet_replica.cc:261] Starting instrumentation
1214 20:52:30.876370 (+ 137us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":279159,"raft.run_cpu_time_us":49,"raft.run_wall_time_us":49,"spinlock_wait_cycles":1792,"tablet-open.queue_time_us":14931980,"thread_start_us":444694,"threads_started":1}
I20251214 20:52:31.449430 1202 tablet_bootstrap.cc:492] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:31.451210 850 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.010s sys 0.025s
W20251214 20:52:31.451339 850 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.010s sys 0.025s
I20251214 20:52:31.455053 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d424e3387698434f92e84eb1d43f5949" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:31.456657 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc73d590531c409e9f60670435dc8223" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:31.457116 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b47ca1a03944cf1847d3578856dafa7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:31.457535 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ebfc373fbf41b1920c271a15302484" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:31.457935 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274185a4115b458e8e9a376bdbcc870b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:31.458704 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65e79db74fa046698507c83416ee1669" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:31.459111 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b69bb4d5979f492683fc13238a668b9f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:31.460870 991 leader_election.cc:343] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.461035 991 leader_election.cc:304] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.461445 2735 raft_consensus.cc:2749] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:31.462731 991 leader_election.cc:343] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.462841 991 leader_election.cc:304] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.463205 2735 raft_consensus.cc:2749] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:31.463385 848 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.572s user 0.009s sys 0.026s
I20251214 20:52:31.467680 2777 raft_consensus.cc:493] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.467849 2777 raft_consensus.cc:515] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.468412 2777 leader_election.cc:290] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:31.473681 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a23561b9f4e48bbbd796e14fafec086" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:31.476948 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25dcf491653147e7831217c1ec7a2e0c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:31.484396 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a23561b9f4e48bbbd796e14fafec086" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:31.484627 1056 raft_consensus.cc:2468] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:31.487244 991 leader_election.cc:343] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.487257 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 423) took 3319 ms. Trace:
I20251214 20:52:31.487397 991 leader_election.cc:304] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.487377 1977 rpcz_store.cc:276] 1214 20:52:28.167485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.169397 (+ 1912us) service_pool.cc:225] Handling call
1214 20:52:30.881845 (+2712448us) ts_tablet_manager.cc:1151] Deleting tablet d13f93bb5e5f4b59bcf9f4bfed070f25
1214 20:52:30.881857 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:31.181729 (+299872us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:31.389482 (+207753us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:31.487232 (+ 97750us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2712309}
I20251214 20:52:31.487742 1977 tablet_replica.cc:333] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:31.487975 1977 raft_consensus.cc:2243] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:31.488092 2735 raft_consensus.cc:2749] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:31.489275 991 leader_election.cc:343] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.489383 991 leader_election.cc:304] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.489439 736 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37360 (request call id 251) took 1628 ms. Trace:
I20251214 20:52:31.489530 736 rpcz_store.cc:276] 1214 20:52:29.861124 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.865996 (+ 4872us) service_pool.cc:225] Handling call
1214 20:52:30.800176 (+934180us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x616000196954. stack: 00007f95284c1ee6 00007f95284c1ca0 00007f952749d3ad 0000000000984f94 0000000000984f5d 000000000098492a 00007f953437a2c0 00007f953822ab2f 00007f953264fc36 00007f953264f965 00007f952f48056e 00007f952f47f44e 00007f952f485685 00007f952f485faf 00007f952f485dd8 0000000000a8970c
1214 20:52:31.098371 (+298195us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x61800010388c. stack: 00007f95284c1ee6 00007f95284c1ca0 00007f952749d3ad 0000000000984f94 0000000000984f5d 000000000098492a 00007f953590744e 00007f9535703091 00007f953822a938 00007f953264fc36 00007f953264f965 00007f952f48056e 00007f952f47f44e 00007f952f485685 00007f952f485faf 00007f952f485dd8
1214 20:52:31.489383 (+391012us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":668215936}
I20251214 20:52:31.489995 2735 raft_consensus.cc:2749] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:31.490224 1202 tablet_bootstrap.cc:654] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:31.490556 991 leader_election.cc:343] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.490649 991 leader_election.cc:304] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.491065 2735 raft_consensus.cc:2749] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:31.492770 991 leader_election.cc:343] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.492872 991 leader_election.cc:304] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.493295 2735 raft_consensus.cc:2749] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:31.493743 991 leader_election.cc:343] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.493836 991 leader_election.cc:304] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.494225 2735 raft_consensus.cc:2749] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:31.498354 733 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37360 (request call id 255) took 1093 ms. Trace:
I20251214 20:52:31.498529 733 rpcz_store.cc:276] 1214 20:52:30.404880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.404941 (+ 61us) service_pool.cc:225] Handling call
1214 20:52:30.805117 (+400176us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x616000196954. stack: 00007f95284c1ee6 00007f95284c1ca0 00007f952749d3ad 0000000000984f94 0000000000984f5d 000000000098492a 00007f953437a2c0 00007f953822ab2f 00007f953264fc36 00007f953264f965 00007f952f48056e 00007f952f47f44e 00007f952f485685 00007f952f485faf 00007f952f485dd8 0000000000a8970c
1214 20:52:31.498310 (+693193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":437227392}
I20251214 20:52:31.499195 1203 tablet_bootstrap.cc:492] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.500008 1203 ts_tablet_manager.cc:1403] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.012s
I20251214 20:52:31.500876 1203 raft_consensus.cc:359] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.501134 1203 raft_consensus.cc:385] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.501236 1203 raft_consensus.cc:740] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.501490 1203 consensus_queue.cc:260] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.502121 1203 ts_tablet_manager.cc:1434] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:31.502938 1203 tablet_bootstrap.cc:492] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.504570 1203 tablet_bootstrap.cc:654] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.507927 756 tablet_service.cc:1505] Processing CreateTablet for tablet ddd625b1cf4a4af191ad4e9a85f7b976 (DEFAULT_TABLE table=default.Table_58bb2f3098084d73bafbd51cde268a21 [id=1e84b30c44fc49ba8a8b35c7e3252761]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.509693 1206 raft_consensus.cc:359] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.510075 1206 raft_consensus.cc:385] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.510178 1206 raft_consensus.cc:740] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.510455 1206 consensus_queue.cc:260] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.511091 1206 ts_tablet_manager.cc:1434] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.357s user 0.000s sys 0.002s
I20251214 20:52:31.514710 2735 raft_consensus.cc:493] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.514874 2735 raft_consensus.cc:515] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.515602 2735 leader_election.cc:290] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.517793 1017 tablet_service.cc:1505] Processing CreateTablet for tablet cdd2a91f9d4c44548966f8e68a75d8de (DEFAULT_TABLE table=default.Table_1aadd2d8580e419fb5a3767b09353c25 [id=e4d01eb0c006487bb2f14fb3e14dd18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.518342 1977 pending_rounds.cc:70] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:31.518415 1977 pending_rounds.cc:77] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493766224400384 op_type: NO_OP noop_request { }
I20251214 20:52:31.518514 1977 raft_consensus.cc:2889] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:31.518610 1977 raft_consensus.cc:2272] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:31.519291 1977 ts_tablet_manager.cc:1916] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:31.514514 1023 tablet_service.cc:1505] Processing CreateTablet for tablet b51d8e5dc7e04fdb9c9d3180c1fe9bdb (DEFAULT_TABLE table=default.Table_09270706a48c42fea27376b3ef3a0746 [id=3f15c09d26b04d519d92594446cca818]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.520532 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 009081f974344dac83e66cdf2294ceac (DEFAULT_TABLE table=default.Table_2fca6840abf741ed94baef488b3640dc [id=eb416f50656b403e98245036103dcdba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.521411 754 tablet_service.cc:1505] Processing CreateTablet for tablet 1dc5e36369574906a0602145b37aa8d6 (DEFAULT_TABLE table=default.Table_da61d1abe3f24498977b6cc43d2ed4df [id=800f51296fe14db2985b673976f6ec6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.521683 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7353593bda4fceaa389c393ddc19c6" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:31.522274 989 leader_election.cc:343] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.524294 2703 raft_consensus.cc:493] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.525265 2703 raft_consensus.cc:515] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.525902 2703 leader_election.cc:290] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.529488 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7353593bda4fceaa389c393ddc19c6" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:31.529606 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feb19ea389af4ea6bb158a0e8aaeb094" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:31.530162 989 leader_election.cc:343] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.530789 1203 tablet_bootstrap.cc:492] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.530766 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feb19ea389af4ea6bb158a0e8aaeb094" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:31.530961 1203 ts_tablet_manager.cc:1403] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
W20251214 20:52:31.531318 991 leader_election.cc:343] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.531431 991 leader_election.cc:304] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.532092 2735 raft_consensus.cc:2749] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:31.532711 1203 raft_consensus.cc:359] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.532968 1203 raft_consensus.cc:385] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.533077 1203 raft_consensus.cc:740] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.533349 1203 consensus_queue.cc:260] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.533947 1203 ts_tablet_manager.cc:1434] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251214 20:52:31.534320 1203 tablet_bootstrap.cc:492] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:31.534636 991 leader_election.cc:343] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.534749 991 leader_election.cc:304] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.535248 2735 raft_consensus.cc:2749] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:31.535323 752 tablet_service.cc:1505] Processing CreateTablet for tablet 6f231db5cc4a4060a996465defb23c6c (DEFAULT_TABLE table=default.Table_e277c40da9e24b7fabb8edd25ec3413d [id=e973536efad243e5b94f5994d57325b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.536437 745 tablet_service.cc:1505] Processing CreateTablet for tablet 009081f974344dac83e66cdf2294ceac (DEFAULT_TABLE table=default.Table_2fca6840abf741ed94baef488b3640dc [id=eb416f50656b403e98245036103dcdba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.538183 1203 tablet_bootstrap.cc:654] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:31.539965 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 56301.3us
W20251214 20:52:31.541299 848 consensus_peers.cc:597] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:31.541635 848 consensus_peers.cc:597] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:31.541800 1202 tablet_bootstrap.cc:492] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:31.541988 1202 ts_tablet_manager.cc:1403] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.093s user 0.000s sys 0.003s
W20251214 20:52:31.542673 848 leader_election.cc:343] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.542737 1202 raft_consensus.cc:359] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.542959 1202 raft_consensus.cc:385] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.543021 1202 raft_consensus.cc:740] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.543228 1202 consensus_queue.cc:260] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
W20251214 20:52:31.543687 848 consensus_peers.cc:597] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:31.543968 848 leader_election.cc:343] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.544332 1203 tablet_bootstrap.cc:492] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.544518 1203 ts_tablet_manager.cc:1403] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20251214 20:52:31.545338 1203 raft_consensus.cc:359] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.545570 1203 raft_consensus.cc:385] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.545666 1203 raft_consensus.cc:740] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.546010 1203 consensus_queue.cc:260] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.543721 1202 ts_tablet_manager.cc:1434] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:31.546749 1203 ts_tablet_manager.cc:1434] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:31.546931 1202 tablet_bootstrap.cc:492] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:31.547246 1203 tablet_bootstrap.cc:492] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.547933 851 leader_election.cc:304] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:31.548298 2777 raft_consensus.cc:2804] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:31.548542 2777 raft_consensus.cc:697] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:31.548835 2777 consensus_queue.cc:237] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.549069 1203 tablet_bootstrap.cc:654] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.550518 746 tablet_service.cc:1505] Processing CreateTablet for tablet b51d8e5dc7e04fdb9c9d3180c1fe9bdb (DEFAULT_TABLE table=default.Table_09270706a48c42fea27376b3ef3a0746 [id=3f15c09d26b04d519d92594446cca818]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.553376 1202 tablet_bootstrap.cc:654] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:31.553745 851 leader_election.cc:343] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 25dcf491653147e7831217c1ec7a2e0c
I20251214 20:52:31.553898 851 leader_election.cc:304] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:31.554162 2777 raft_consensus.cc:2749] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:31.554549 851 consensus_peers.cc:597] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18e119bc44874aecb4e83c8d6040c8cd. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:31.555496 1203 tablet_bootstrap.cc:492] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.555703 1203 ts_tablet_manager.cc:1403] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20251214 20:52:31.555727 2777 consensus_queue.cc:1048] T 876d2a1f1d1c4f6882a5857b43568ee5 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:31.556784 851 leader_election.cc:304] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:31.557031 2777 raft_consensus.cc:2804] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:31.557132 2777 raft_consensus.cc:493] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:31.557212 2777 raft_consensus.cc:3060] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:31.557477 1203 raft_consensus.cc:359] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.557706 1203 raft_consensus.cc:385] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.557796 1203 raft_consensus.cc:740] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.558079 1203 consensus_queue.cc:260] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.558713 1203 ts_tablet_manager.cc:1434] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251214 20:52:31.559064 1203 tablet_bootstrap.cc:492] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.560894 753 tablet_service.cc:1505] Processing CreateTablet for tablet cdd2a91f9d4c44548966f8e68a75d8de (DEFAULT_TABLE table=default.Table_1aadd2d8580e419fb5a3767b09353c25 [id=e4d01eb0c006487bb2f14fb3e14dd18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.561285 1203 tablet_bootstrap.cc:654] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.561621 755 tablet_service.cc:1505] Processing CreateTablet for tablet f6b7fe43fa104199be5363ca53035121 (DEFAULT_TABLE table=default.Table_1225d07e2e7a4aa1b60557b64de1f35e [id=07f7bb3503114ba3b0e1fe404826d582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.567047 1203 tablet_bootstrap.cc:492] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.567423 1203 ts_tablet_manager.cc:1403] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20251214 20:52:31.568693 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 1def5f0de5ce4eb0bed36f626c4f0683 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:31.568653 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 348d8dcef97042279bc0d5ff31d3c137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:31.569056 876 tablet_service.cc:1552] Processing DeleteTablet for tablet 06381efe75284e649fd6d1a9903cfa9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:18 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:31.569216 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e813373fbd482298312ead4f037c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:31.569537 881 tablet_service.cc:1505] Processing CreateTablet for tablet 234a3f2c04e94149a1aeca848e6a4ce4 (DEFAULT_TABLE table=default.Table_bdfed01cc4cd4458be399de18a738f45 [id=164c3a6291b644789ae5905faca45b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.569602 876 tablet_service.cc:1505] Processing CreateTablet for tablet ddd625b1cf4a4af191ad4e9a85f7b976 (DEFAULT_TABLE table=default.Table_58bb2f3098084d73bafbd51cde268a21 [id=1e84b30c44fc49ba8a8b35c7e3252761]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.569911 879 tablet_service.cc:1505] Processing CreateTablet for tablet 1dc5e36369574906a0602145b37aa8d6 (DEFAULT_TABLE table=default.Table_da61d1abe3f24498977b6cc43d2ed4df [id=800f51296fe14db2985b673976f6ec6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.568418 1203 raft_consensus.cc:359] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.570703 1203 raft_consensus.cc:385] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.570814 1203 raft_consensus.cc:740] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.570863 1202 tablet_bootstrap.cc:492] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:31.571002 1202 ts_tablet_manager.cc:1403] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251214 20:52:31.571254 1203 consensus_queue.cc:260] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.571738 1202 raft_consensus.cc:359] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:31.571916 1202 raft_consensus.cc:385] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.571979 1202 raft_consensus.cc:740] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.572121 1203 ts_tablet_manager.cc:1434] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:31.572197 1202 consensus_queue.cc:260] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:31.572706 1202 ts_tablet_manager.cc:1434] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:31.572813 1203 tablet_bootstrap.cc:492] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.572988 1202 tablet_bootstrap.cc:492] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:31.574535 1202 tablet_bootstrap.cc:654] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.574918 1203 tablet_bootstrap.cc:654] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:31.577630 712 consensus_peers.cc:597] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Couldn't send request to peer a2e8fd1cb731446d88f8e510c7a52999. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:31.586616 2791 raft_consensus.cc:493] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.586802 2791 raft_consensus.cc:515] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.587572 2791 leader_election.cc:290] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:31.588222 2735 raft_consensus.cc:493] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.588374 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f2314d33fa474ca36dafe0a0eae5ae" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:31.588367 2735 raft_consensus.cc:515] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.588575 773 raft_consensus.cc:2468] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:31.588969 2735 leader_election.cc:290] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.589018 848 leader_election.cc:304] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:31.589259 2791 raft_consensus.cc:2804] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 20:52:31.589326 2791 raft_consensus.cc:493] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:31.589375 2791 raft_consensus.cc:3060] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Advancing to term 2
I20251214 20:52:31.589892 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f11834cabdad445b8e53cd237d146282" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:31.590497 991 leader_election.cc:343] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.591058 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f11834cabdad445b8e53cd237d146282" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:31.591531 989 leader_election.cc:343] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.591636 735 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37360 (request call id 254) took 1210 ms. Trace:
I20251214 20:52:31.591634 989 leader_election.cc:304] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.591713 735 rpcz_store.cc:276] 1214 20:52:30.381259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.381415 (+ 156us) service_pool.cc:225] Handling call
1214 20:52:30.804926 (+423511us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x616000196954. stack: 00007f95284c1ee6 00007f95284c1ca0 00007f952749d3ad 0000000000984f94 0000000000984f5d 000000000098492a 00007f953437a2c0 00007f953822ab2f 00007f953264fc36 00007f953264f965 00007f952f48056e 00007f952f47f44e 00007f952f485685 00007f952f485faf 00007f952f485dd8 0000000000a8970c
1214 20:52:31.055274 (+250348us) spinlock_profiling.cc:243] Waited 32.3 ms on lock 0x60c0002748b8. stack: 00007f95284c1ee6 00007f95284c1ca0 00007f952749d3ad 0000000000984f94 0000000000984f5d 000000000098492a 00007f9533c905fb 00007f9533c4d8c3 00007f9535703f28 00007f95357033ea 00007f953822a938 00007f953264fc36 00007f953264f965 00007f952f48056e 00007f952f47f44e 00007f952f485685
1214 20:52:31.591591 (+536317us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":536145536}
I20251214 20:52:31.592260 2735 raft_consensus.cc:2749] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:31.594074 851 consensus_peers.cc:597] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dca369dc5f54a4ea1783e334c5670d6. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:31.594478 851 leader_election.cc:343] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 57f2314d33fa474ca36dafe0a0eae5ae
I20251214 20:52:31.595665 1202 tablet_bootstrap.cc:492] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:31.595808 1202 ts_tablet_manager.cc:1403] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20251214 20:52:31.596489 1202 raft_consensus.cc:359] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.596666 1202 raft_consensus.cc:385] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.596730 1202 raft_consensus.cc:740] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.596941 1202 consensus_queue.cc:260] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.597419 1202 ts_tablet_manager.cc:1434] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:31.597719 1202 tablet_bootstrap.cc:492] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:31.599418 1202 tablet_bootstrap.cc:654] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.600764 2786 raft_consensus.cc:493] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.600946 2786 raft_consensus.cc:515] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.601589 2786 leader_election.cc:290] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:31.603569 851 leader_election.cc:343] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: f84e9dfc23d34eb0a379e8ac5a442e59
I20251214 20:52:31.604049 1206 tablet_bootstrap.cc:492] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:31.607079 1206 tablet_bootstrap.cc:654] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.614421 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd2a91f9d4c44548966f8e68a75d8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.614715 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b51d8e5dc7e04fdb9c9d3180c1fe9bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.614877 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009081f974344dac83e66cdf2294ceac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.616590 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234a3f2c04e94149a1aeca848e6a4ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.617214 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddd625b1cf4a4af191ad4e9a85f7b976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.617624 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc5e36369574906a0602145b37aa8d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.620761 2777 raft_consensus.cc:515] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.623567 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddd625b1cf4a4af191ad4e9a85f7b976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.623893 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc5e36369574906a0602145b37aa8d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.624464 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009081f974344dac83e66cdf2294ceac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.624963 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f231db5cc4a4060a996465defb23c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.625510 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b51d8e5dc7e04fdb9c9d3180c1fe9bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.626403 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6b7fe43fa104199be5363ca53035121. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.638098 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd2a91f9d4c44548966f8e68a75d8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.641086 2747 raft_consensus.cc:493] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2e8fd1cb731446d88f8e510c7a52999)
I20251214 20:52:31.641311 2747 raft_consensus.cc:515] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.642072 2747 leader_election.cc:290] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.642656 2777 leader_election.cc:290] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:31.644723 712 leader_election.cc:343] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: ec85857f047e44d78835305ddcb70768
I20251214 20:52:31.656818 2791 raft_consensus.cc:515] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.658128 2791 leader_election.cc:290] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:31.658526 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f84e9dfc23d34eb0a379e8ac5a442e59" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:31.659735 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a23561b9f4e48bbbd796e14fafec086" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:31.661334 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57f2314d33fa474ca36dafe0a0eae5ae" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:31.661499 773 raft_consensus.cc:3060] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Advancing to term 2
W20251214 20:52:31.665901 848 leader_election.cc:343] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.666749 848 leader_election.cc:304] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:31.668676 848 leader_election.cc:343] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.671062 2791 raft_consensus.cc:2749] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:31.676931 1977 ts_tablet_manager.cc:1929] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:31.677273 1977 log.cc:1199] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/58ac06f852fc4abea5cb213926002ca1
I20251214 20:52:31.679256 1977 ts_tablet_manager.cc:1950] T 58ac06f852fc4abea5cb213926002ca1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:31.625550 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a23561b9f4e48bbbd796e14fafec086" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:31.689684 1051 raft_consensus.cc:3060] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:31.690332 1202 tablet_bootstrap.cc:492] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:31.690692 1202 ts_tablet_manager.cc:1403] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.093s user 0.040s sys 0.033s
W20251214 20:52:31.691047 851 leader_election.cc:343] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 57f2314d33fa474ca36dafe0a0eae5ae
W20251214 20:52:31.691457 713 leader_election.cc:343] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: ec85857f047e44d78835305ddcb70768
I20251214 20:52:31.691589 713 leader_election.cc:304] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.692052 2747 raft_consensus.cc:2749] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:31.692945 1202 raft_consensus.cc:359] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.693351 1202 raft_consensus.cc:385] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.693481 1202 raft_consensus.cc:740] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.694653 1202 consensus_queue.cc:260] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.698556 1202 ts_tablet_manager.cc:1434] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.008s user 0.001s sys 0.003s
I20251214 20:52:31.701258 1202 tablet_bootstrap.cc:492] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:31.714357 1202 tablet_bootstrap.cc:654] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.714427 879 tablet_service.cc:1505] Processing CreateTablet for tablet 6f231db5cc4a4060a996465defb23c6c (DEFAULT_TABLE table=default.Table_e277c40da9e24b7fabb8edd25ec3413d [id=e973536efad243e5b94f5994d57325b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.715494 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f231db5cc4a4060a996465defb23c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.721488 881 tablet_service.cc:1505] Processing CreateTablet for tablet 009081f974344dac83e66cdf2294ceac (DEFAULT_TABLE table=default.Table_2fca6840abf741ed94baef488b3640dc [id=eb416f50656b403e98245036103dcdba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.721968 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009081f974344dac83e66cdf2294ceac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.727973 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]) successfully deleted
I20251214 20:52:31.728909 876 tablet_service.cc:1505] Processing CreateTablet for tablet b51d8e5dc7e04fdb9c9d3180c1fe9bdb (DEFAULT_TABLE table=default.Table_09270706a48c42fea27376b3ef3a0746 [id=3f15c09d26b04d519d92594446cca818]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.729398 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b51d8e5dc7e04fdb9c9d3180c1fe9bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.732051 1202 tablet_bootstrap.cc:492] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:31.732204 1202 ts_tablet_manager.cc:1403] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20251214 20:52:31.732957 1202 raft_consensus.cc:359] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.733129 1202 raft_consensus.cc:385] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.733172 1202 raft_consensus.cc:740] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.733362 1202 consensus_queue.cc:260] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.733806 1202 ts_tablet_manager.cc:1434] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:31.734228 1202 tablet_bootstrap.cc:492] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:31.737852 773 raft_consensus.cc:2468] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 2.
I20251214 20:52:31.738569 848 leader_election.cc:304] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:31.738905 1206 tablet_bootstrap.cc:492] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:31.739080 1206 ts_tablet_manager.cc:1403] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.001s
I20251214 20:52:31.739806 1206 raft_consensus.cc:359] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.739966 1206 raft_consensus.cc:385] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.740025 1206 raft_consensus.cc:740] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.740489 2787 raft_consensus.cc:2804] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 20:52:31.740629 2787 raft_consensus.cc:697] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 2 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:31.740896 2787 consensus_queue.cc:237] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.744519 2771 raft_consensus.cc:493] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.744697 2771 raft_consensus.cc:515] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.745476 2771 leader_election.cc:290] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:31.745558 2735 raft_consensus.cc:493] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.745705 2735 raft_consensus.cc:515] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.746271 2735 leader_election.cc:290] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.747255 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1eb4c3aa654480caaa69dbd5047365c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:31.747789 991 leader_election.cc:343] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:31.747984 851 leader_election.cc:343] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 492770a7b8b74ba49315f974cc54eedb
I20251214 20:52:31.750533 1202 tablet_bootstrap.cc:654] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.764943 894 tablet_service.cc:1505] Processing CreateTablet for tablet cdd2a91f9d4c44548966f8e68a75d8de (DEFAULT_TABLE table=default.Table_1aadd2d8580e419fb5a3767b09353c25 [id=e4d01eb0c006487bb2f14fb3e14dd18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.765544 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd2a91f9d4c44548966f8e68a75d8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.747656 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "492770a7b8b74ba49315f974cc54eedb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:31.748306 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1eb4c3aa654480caaa69dbd5047365c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:31.773327 989 leader_election.cc:343] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.773451 989 leader_election.cc:304] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.774238 2735 raft_consensus.cc:2749] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:31.774641 848 leader_election.cc:343] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.774818 848 leader_election.cc:304] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:31.740265 1206 consensus_queue.cc:260] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.775492 2791 raft_consensus.cc:2749] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:31.783856 2816 raft_consensus.cc:493] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c27224391721467a8f46e467a980b37a)
I20251214 20:52:31.784178 2816 raft_consensus.cc:515] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:31.785223 2816 leader_election.cc:290] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:31.798614 1334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" is_pre_election: true
I20251214 20:52:31.798936 1334 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6713a20a82b434f8a8bc122113e26ad in term 4.
I20251214 20:52:31.786424 463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a" is_pre_election: true
W20251214 20:52:31.801354 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 73431.7us
I20251214 20:52:31.806833 1206 ts_tablet_manager.cc:1434] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.068s user 0.002s sys 0.000s
I20251214 20:52:31.813534 1206 tablet_bootstrap.cc:492] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:31.814641 2777 raft_consensus.cc:493] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.814980 2777 raft_consensus.cc:515] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:31.817708 851 leader_election.cc:343] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 69f6108792fa4c2a9744e3248c76251f
I20251214 20:52:31.818032 2777 leader_election.cc:290] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:31.817231 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69f6108792fa4c2a9744e3248c76251f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:31.827404 848 leader_election.cc:343] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.827586 848 leader_election.cc:304] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:31.828214 2787 raft_consensus.cc:2749] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:31.829727 2291 leader_election.cc:304] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, e6713a20a82b434f8a8bc122113e26ad; no voters: c27224391721467a8f46e467a980b37a
I20251214 20:52:31.831437 1206 tablet_bootstrap.cc:654] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.833544 2823 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c27224391721467a8f46e467a980b37a)
I20251214 20:52:31.834383 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 425) took 3666 ms. Trace:
I20251214 20:52:31.834545 1977 rpcz_store.cc:276] 1214 20:52:28.167676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.169558 (+ 1882us) service_pool.cc:225] Handling call
1214 20:52:31.487676 (+3318118us) ts_tablet_manager.cc:1151] Deleting tablet 58ac06f852fc4abea5cb213926002ca1
1214 20:52:31.487686 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:31.616487 (+128801us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:31.676698 (+ 60211us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:31.834350 (+157652us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30170,"tablet-delete.queue_time_us":3317985}
I20251214 20:52:31.838274 2823 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:31.841130 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 58ac06f852fc4abea5cb213926002ca1 (table default.Table_3369f89218b140e7a52da8221726f4e1 [id=e901e2802c854b4f829e59e00aa90040]) successfully deleted
W20251214 20:52:31.844100 2779 thread.cc:641] leader-initialization [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.005s sys 0.003s
W20251214 20:52:31.844774 2779 thread.cc:608] leader-initialization [worker] (thread pool) Time spent starting thread: real 0.837s user 0.005s sys 0.003s
I20251214 20:52:31.846729 893 tablet_service.cc:1505] Processing CreateTablet for tablet 078dff96c9704e36b6b26692f51b890e (DEFAULT_TABLE table=default.Table_176fae22b3bf4a159ac799ec6339ea16 [id=08bd7456e0db4786a4f2508d2709abea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.846952 878 tablet_service.cc:1505] Processing CreateTablet for tablet 37d198fa8da642dca442476578a35f1e (DEFAULT_TABLE table=default.Table_81b328890d6748ea8a95ac3cd696c9dd [id=4a964e807c8a4161bf9fa9cb732ba21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.847579 892 tablet_service.cc:1505] Processing CreateTablet for tablet 707393e4f91e4b1dbe1cf7485fa65624 (DEFAULT_TABLE table=default.Table_17c9057bb25b4cce9b1072a93f6ea38a [id=5727ded20a714c31be1ba62f2dfb21e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.847761 752 tablet_service.cc:1505] Processing CreateTablet for tablet 6343dbb0658c4145a816c444ff5297d1 (DEFAULT_TABLE table=default.Table_8d0c148da21f4137b09c9631e0b3f9bf [id=fa40006b630d491e88479cd9f61def2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.847963 746 tablet_service.cc:1505] Processing CreateTablet for tablet 707393e4f91e4b1dbe1cf7485fa65624 (DEFAULT_TABLE table=default.Table_17c9057bb25b4cce9b1072a93f6ea38a [id=5727ded20a714c31be1ba62f2dfb21e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.848232 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6343dbb0658c4145a816c444ff5297d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.848439 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707393e4f91e4b1dbe1cf7485fa65624. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.852150 755 tablet_service.cc:1505] Processing CreateTablet for tablet 078dff96c9704e36b6b26692f51b890e (DEFAULT_TABLE table=default.Table_176fae22b3bf4a159ac799ec6339ea16 [id=08bd7456e0db4786a4f2508d2709abea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.852545 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 078dff96c9704e36b6b26692f51b890e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.861792 1203 tablet_bootstrap.cc:492] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.861972 1203 ts_tablet_manager.cc:1403] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.289s user 0.024s sys 0.076s
I20251214 20:52:31.862471 885 tablet_service.cc:1505] Processing CreateTablet for tablet 6343dbb0658c4145a816c444ff5297d1 (DEFAULT_TABLE table=default.Table_8d0c148da21f4137b09c9631e0b3f9bf [id=fa40006b630d491e88479cd9f61def2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.865235 1203 raft_consensus.cc:359] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.865440 1203 raft_consensus.cc:385] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.865496 1203 raft_consensus.cc:740] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.865694 1203 consensus_queue.cc:260] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.872615 894 tablet_service.cc:1505] Processing CreateTablet for tablet f6b7fe43fa104199be5363ca53035121 (DEFAULT_TABLE table=default.Table_1225d07e2e7a4aa1b60557b64de1f35e [id=07f7bb3503114ba3b0e1fe404826d582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.872818 887 tablet_service.cc:1505] Processing CreateTablet for tablet 438b8ec5e9834442994966c8d83c1a31 (DEFAULT_TABLE table=default.Table_5cffb2d16a4847caa5cf6a2c4c238da8 [id=a8b3bbe26fc14b4bbc8a78c3d1f9df6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.873800 752 tablet_service.cc:1505] Processing CreateTablet for tablet 438b8ec5e9834442994966c8d83c1a31 (DEFAULT_TABLE table=default.Table_5cffb2d16a4847caa5cf6a2c4c238da8 [id=a8b3bbe26fc14b4bbc8a78c3d1f9df6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.874325 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438b8ec5e9834442994966c8d83c1a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.879325 753 tablet_service.cc:1505] Processing CreateTablet for tablet 37d198fa8da642dca442476578a35f1e (DEFAULT_TABLE table=default.Table_81b328890d6748ea8a95ac3cd696c9dd [id=4a964e807c8a4161bf9fa9cb732ba21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.884467 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 428) took 3694 ms. Trace:
I20251214 20:52:31.884631 1977 rpcz_store.cc:276] 1214 20:52:28.190055 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.190147 (+ 92us) service_pool.cc:225] Handling call
1214 20:52:31.834953 (+3644806us) ts_tablet_manager.cc:1151] Deleting tablet 0a2fbec0543445b7856ef22c934c6840
1214 20:52:31.834964 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:31.884436 (+ 49472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3644505}
I20251214 20:52:31.885340 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 078dff96c9704e36b6b26692f51b890e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:31.885317 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 0a2fbec0543445b7856ef22c934c6840 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2fbec0543445b7856ef22c934c6840 already in progress: creating tablet
I20251214 20:52:31.886530 463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a" is_pre_election: true
I20251214 20:52:31.886646 2823 leader_election.cc:290] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819)
I20251214 20:52:31.887385 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707393e4f91e4b1dbe1cf7485fa65624. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.892314 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438b8ec5e9834442994966c8d83c1a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.896546 1206 tablet_bootstrap.cc:492] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:31.896837 1206 ts_tablet_manager.cc:1403] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.000s
I20251214 20:52:31.897648 1206 raft_consensus.cc:359] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:31.897899 2816 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251214 20:52:31.898098 2816 raft_consensus.cc:493] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 4 FOLLOWER]: Starting leader election (detected failure of leader c27224391721467a8f46e467a980b37a)
I20251214 20:52:31.898218 2816 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 4 FOLLOWER]: Advancing to term 5
I20251214 20:52:31.899483 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d198fa8da642dca442476578a35f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.903424 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6343dbb0658c4145a816c444ff5297d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.906864 2777 raft_consensus.cc:493] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.907083 2777 raft_consensus.cc:515] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.907423 1203 ts_tablet_manager.cc:1434] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20251214 20:52:31.907876 1203 tablet_bootstrap.cc:492] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.908421 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1f4ea72c6274569a98cab6afdc55d55" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:31.909370 1202 tablet_bootstrap.cc:492] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:31.909740 1202 ts_tablet_manager.cc:1403] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.176s user 0.021s sys 0.041s
W20251214 20:52:31.910413 851 leader_election.cc:343] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Illegal state: Tablet not RUNNING: STOPPED
I20251214 20:52:31.910687 2777 leader_election.cc:290] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:31.911232 1202 raft_consensus.cc:359] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.911412 1202 raft_consensus.cc:385] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.911474 1202 raft_consensus.cc:740] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.911674 1202 consensus_queue.cc:260] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.912153 1202 ts_tablet_manager.cc:1434] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:31.912423 1202 tablet_bootstrap.cc:492] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:31.913933 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 429) took 3643 ms. Trace:
I20251214 20:52:31.914026 1977 rpcz_store.cc:276] 1214 20:52:28.270216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.270270 (+ 54us) service_pool.cc:225] Handling call
1214 20:52:31.884986 (+3614716us) ts_tablet_manager.cc:1151] Deleting tablet 088c72a6f5014b6fbc6ee060b6b7a590
1214 20:52:31.913778 (+ 28792us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:31.913910 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3604836}
I20251214 20:52:31.914333 1977 tablet_replica.cc:333] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:31.914546 1977 raft_consensus.cc:2243] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:31.914886 1977 raft_consensus.cc:2272] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:31.914839 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 088c72a6f5014b6fbc6ee060b6b7a590 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 088c72a6f5014b6fbc6ee060b6b7a590 already in progress: creating tablet
I20251214 20:52:31.915516 1977 ts_tablet_manager.cc:1916] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:31.918838 1206 raft_consensus.cc:385] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.918987 1206 raft_consensus.cc:740] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.899407 740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57406 (request call id 515) took 1159 ms. Trace:
I20251214 20:52:31.908947 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1f4ea72c6274569a98cab6afdc55d55" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:31.919275 1206 consensus_queue.cc:260] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:31.919759 1206 ts_tablet_manager.cc:1434] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
W20251214 20:52:31.919754 848 leader_election.cc:343] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.919860 848 leader_election.cc:304] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:31.922487 1206 tablet_bootstrap.cc:492] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:31.925447 2777 raft_consensus.cc:2749] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:31.927376 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6b7fe43fa104199be5363ca53035121. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.928748 740 rpcz_store.cc:276] 1214 20:52:30.740195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.740287 (+ 92us) service_pool.cc:225] Handling call
1214 20:52:30.803119 (+ 62832us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1214 20:52:30.803185 (+ 66us) ts_tablet_manager.cc:682] Creating new metadata...
1214 20:52:31.834499 (+1031314us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:31.899233 (+ 64734us) tablet_replica.cc:206] Creating consensus instance
1214 20:52:31.899390 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5888}
I20251214 20:52:31.929457 2735 raft_consensus.cc:493] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.929600 2735 raft_consensus.cc:515] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.930233 2735 leader_election.cc:290] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:31.930893 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1d0e555eee4c56be271d14c34335bd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:31.931547 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1d0e555eee4c56be271d14c34335bd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:31.932092 991 leader_election.cc:343] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.932240 1203 tablet_bootstrap.cc:654] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:31.932478 989 leader_election.cc:343] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:31.932580 989 leader_election.cc:304] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:31.932996 2735 raft_consensus.cc:2749] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:31.936049 455 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a12e8053c65343758d518ee840dbaee8 consensus state has no leader
I20251214 20:52:31.939877 1051 raft_consensus.cc:2468] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:31.940008 1206 tablet_bootstrap.cc:654] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.940416 1202 tablet_bootstrap.cc:654] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.940549 851 leader_election.cc:304] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:31.941778 2777 raft_consensus.cc:2804] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:31.941990 2777 raft_consensus.cc:697] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:31.942312 2777 consensus_queue.cc:237] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:31.945313 1203 tablet_bootstrap.cc:492] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:31.945474 1203 ts_tablet_manager.cc:1403] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20251214 20:52:31.946223 1203 raft_consensus.cc:359] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.946412 1203 raft_consensus.cc:385] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.946468 1203 raft_consensus.cc:740] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.946703 1203 consensus_queue.cc:260] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.946780 1856 ts_tablet_manager.cc:1929] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:31.946892 1856 log.cc:1199] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/b1f4ea72c6274569a98cab6afdc55d55
I20251214 20:52:31.947381 1856 ts_tablet_manager.cc:1950] T b1f4ea72c6274569a98cab6afdc55d55 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:31.947440 1982 ts_tablet_manager.cc:1929] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:31.947453 1206 tablet_bootstrap.cc:492] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:31.947809 1206 ts_tablet_manager.cc:1403] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20251214 20:52:31.947839 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d198fa8da642dca442476578a35f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.948593 1206 raft_consensus.cc:359] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.948827 1206 raft_consensus.cc:385] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.948918 1206 raft_consensus.cc:740] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.949178 1206 consensus_queue.cc:260] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.949868 1206 ts_tablet_manager.cc:1434] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:31.950227 1206 tablet_bootstrap.cc:492] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:31.950234 1203 ts_tablet_manager.cc:1434] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:31.950541 1203 tablet_bootstrap.cc:492] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:31.947575 1982 log.cc:1199] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/a38e82f5dd1442f3a1dfcd6fbd94dbaf
I20251214 20:52:31.951915 1206 tablet_bootstrap.cc:654] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.952046 1203 tablet_bootstrap.cc:654] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.964927 1017 tablet_service.cc:1505] Processing CreateTablet for tablet f6b7fe43fa104199be5363ca53035121 (DEFAULT_TABLE table=default.Table_1225d07e2e7a4aa1b60557b64de1f35e [id=07f7bb3503114ba3b0e1fe404826d582]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.965605 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6b7fe43fa104199be5363ca53035121. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.952311 1982 ts_tablet_manager.cc:1950] T a38e82f5dd1442f3a1dfcd6fbd94dbaf P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:31.969009 1206 tablet_bootstrap.cc:492] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:31.969225 1206 ts_tablet_manager.cc:1403] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20251214 20:52:31.969959 1206 raft_consensus.cc:359] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.970196 1206 raft_consensus.cc:385] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.970299 1206 raft_consensus.cc:740] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.970566 1206 consensus_queue.cc:260] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.971423 1206 ts_tablet_manager.cc:1434] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:31.973304 455 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 3a23561b9f4e48bbbd796e14fafec086 (table default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3 [id=f1338e81f4ec4dce87130f2ddbe98041]). Expected version 1 got 0
I20251214 20:52:31.975288 2777 raft_consensus.cc:493] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:31.975495 2777 raft_consensus.cc:515] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.976109 2777 leader_election.cc:290] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:31.976786 1206 tablet_bootstrap.cc:492] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:31.978040 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98b7cac3930a4a9eb16ac413c8f1e092" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:31.978164 1206 tablet_bootstrap.cc:654] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:31.978220 768 raft_consensus.cc:2468] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:31.978685 1202 tablet_bootstrap.cc:492] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:31.978835 1202 ts_tablet_manager.cc:1403] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20251214 20:52:31.978844 848 leader_election.cc:304] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:31.979462 851 leader_election.cc:343] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 98b7cac3930a4a9eb16ac413c8f1e092
I20251214 20:52:31.979736 1202 raft_consensus.cc:359] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.979928 1202 raft_consensus.cc:385] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:31.979983 1202 raft_consensus.cc:740] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:31.980190 1202 consensus_queue.cc:260] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:31.980276 2777 raft_consensus.cc:2804] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251214 20:52:31.980558 2777 raft_consensus.cc:493] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:31.980669 2777 raft_consensus.cc:3060] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Advancing to term 2
I20251214 20:52:31.980720 1202 ts_tablet_manager.cc:1434] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:31.981040 1202 tablet_bootstrap.cc:492] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:31.982010 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 707393e4f91e4b1dbe1cf7485fa65624 (DEFAULT_TABLE table=default.Table_17c9057bb25b4cce9b1072a93f6ea38a [id=5727ded20a714c31be1ba62f2dfb21e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.982383 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707393e4f91e4b1dbe1cf7485fa65624. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.985443 2816 raft_consensus.cc:515] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:31.986315 2816 leader_election.cc:290] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 5 election: Requested vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:31.986871 2397 raft_consensus.cc:1240] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 5 FOLLOWER]: Rejecting Update request from peer c27224391721467a8f46e467a980b37a for earlier term 4. Current term is 5. Ops: []
I20251214 20:52:31.990816 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 6343dbb0658c4145a816c444ff5297d1 (DEFAULT_TABLE table=default.Table_8d0c148da21f4137b09c9631e0b3f9bf [id=fa40006b630d491e88479cd9f61def2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.991266 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6343dbb0658c4145a816c444ff5297d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.994838 2825 consensus_queue.cc:1059] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 }, Status: INVALID_TERM, Last received: 4.955, Next index: 956, Last known committed idx: 945, Time since last communication: 0.000s
I20251214 20:52:31.995446 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 438b8ec5e9834442994966c8d83c1a31 (DEFAULT_TABLE table=default.Table_5cffb2d16a4847caa5cf6a2c4c238da8 [id=a8b3bbe26fc14b4bbc8a78c3d1f9df6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:31.995868 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438b8ec5e9834442994966c8d83c1a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:31.998039 1202 tablet_bootstrap.cc:654] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.000828 2735 raft_consensus.cc:493] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.000979 2735 raft_consensus.cc:515] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.001598 2735 leader_election.cc:290] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.001744 2397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 5 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
I20251214 20:52:32.002204 2397 raft_consensus.cc:2393] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c48b8eaf30c4e2caa94a51d087406d4 in current term 5: Already voted for candidate e6713a20a82b434f8a8bc122113e26ad in this term.
I20251214 20:52:32.002246 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb29fd8c85344ed8aa8d880e2ed240bc" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.002739 1297 leader_election.cc:304] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4; no voters: c27224391721467a8f46e467a980b37a, e6713a20a82b434f8a8bc122113e26ad
W20251214 20:52:32.003360 991 leader_election.cc:343] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.004033 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 078dff96c9704e36b6b26692f51b890e (DEFAULT_TABLE table=default.Table_176fae22b3bf4a159ac799ec6339ea16 [id=08bd7456e0db4786a4f2508d2709abea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:32.004410 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 078dff96c9704e36b6b26692f51b890e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:32.007354 451 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 5b68367f902f4be386788a8d138b5762
I20251214 20:52:32.008021 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb29fd8c85344ed8aa8d880e2ed240bc" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.008620 989 leader_election.cc:343] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.008731 989 leader_election.cc:304] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.008756 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b68367f902f4be386788a8d138b5762 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:32.009564 449 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet 0b7dd0ab78534a34ba342ec6e4fae129
I20251214 20:52:32.009604 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b68367f902f4be386788a8d138b5762 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:32.010491 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b7dd0ab78534a34ba342ec6e4fae129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:32.011170 449 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 515) took 1289 ms. Trace:
I20251214 20:52:32.011281 449 rpcz_store.cc:276] 1214 20:52:30.721071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.869170 (+148099us) service_pool.cc:225] Handling call
1214 20:52:30.936420 (+ 67250us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b1811b6 00007ff411c8c486 00007ff411c8c1b5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:30.951269 (+ 14849us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:30.951428 (+ 159us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:30.951498 (+ 70us) catalog_manager.cc:2824] Locking tablets
1214 20:52:31.107584 (+156086us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:32.009353 (+901769us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:32.009463 (+ 110us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:32.009508 (+ 45us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:32.010074 (+ 566us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:31.113827 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:31.113921 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:31.113925 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:31.113927 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:31.114004 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:31.114011 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:31.114016 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:31.114096 (+ 80us) tablet.cc:699] Row locks acquired
1214 20:52:31.114098 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:31.114121 (+ 23us) write_op.cc:270] Start()
1214 20:52:31.114142 (+ 21us) write_op.cc:276] Timestamp: P: 1765745551114063 usec, L: 0
1214 20:52:31.114145 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:31.114260 (+ 115us) log.cc:844] Serialized 725 byte log entry
1214 20:52:32.003751 (+889491us) op_driver.cc:464] REPLICATION: finished
1214 20:52:32.003822 (+ 71us) write_op.cc:301] APPLY: starting
1214 20:52:32.003832 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:32.003861 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:32.007295 (+ 3434us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:32.007389 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:32.007441 (+ 52us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:32.007449 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:32.007564 (+ 115us) log.cc:844] Serialized 29 byte log entry
1214 20:52:32.007656 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:32.007682 (+ 26us) write_op.cc:454] Released schema lock
1214 20:52:32.007691 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":37298944,"child_traces":[["op",{"apply.queue_time_us":39,"num_ops":2,"prepare.queue_time_us":5524,"prepare.run_cpu_time_us":473,"prepare.run_wall_time_us":472,"replication_time_us":889584}]]}
I20251214 20:52:32.012564 463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "c27224391721467a8f46e467a980b37a"
I20251214 20:52:32.014014 2735 raft_consensus.cc:2749] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.015302 2823 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 4 FOLLOWER]: Advancing to term 5
I20251214 20:52:32.017421 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b7dd0ab78534a34ba342ec6e4fae129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:32.018996 2823 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251214 20:52:32.019186 2735 raft_consensus.cc:493] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.019328 2735 raft_consensus.cc:515] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.019984 2735 leader_election.cc:290] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.019979 452 catalog_manager.cc:6199] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: default.Table_55f303d33ec84a269232fc1ba8f2d182 [id=5ded9fe825ae4308b5bd4b21c16634c8] alter complete (version 1)
I20251214 20:52:32.020622 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "082ec70b8ee646b898d9201ad073d292" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:32.021183 991 leader_election.cc:343] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.021951 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "082ec70b8ee646b898d9201ad073d292" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.022038 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:46855 (request call id 198) took 1049 ms (client timeout 1000 ms). Trace:
W20251214 20:52:32.022151 452 rpcz_store.cc:269] 1214 20:52:30.972417 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.031607 (+ 59190us) service_pool.cc:225] Handling call
1214 20:52:31.074424 (+ 42817us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b17aa08 00007ff411c87ca6 00007ff411c879d5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:32.021957 (+947533us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:31.114331 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:31.114413 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:31.114417 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:31.114419 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:31.114495 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:31.114503 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:31.114508 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:31.114550 (+ 42us) tablet.cc:699] Row locks acquired
1214 20:52:31.114553 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:31.114578 (+ 25us) write_op.cc:270] Start()
1214 20:52:31.114602 (+ 24us) write_op.cc:276] Timestamp: P: 1765745551114519 usec, L: 0
1214 20:52:31.114605 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:31.114728 (+ 123us) log.cc:844] Serialized 356 byte log entry
1214 20:52:32.004061 (+889333us) op_driver.cc:464] REPLICATION: finished
1214 20:52:32.017896 (+ 13835us) write_op.cc:301] APPLY: starting
1214 20:52:32.017919 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:32.017985 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:32.018019 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:32.018102 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:32.018160 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:32.018168 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:32.018292 (+ 124us) log.cc:844] Serialized 23 byte log entry
1214 20:52:32.018387 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:32.018407 (+ 20us) write_op.cc:454] Released schema lock
1214 20:52:32.018413 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":10,"spinlock_wait_cycles":88466304,"child_traces":[["op",{"apply.queue_time_us":13773,"num_ops":1,"prepare.queue_time_us":5756,"prepare.run_cpu_time_us":455,"prepare.run_wall_time_us":453,"replication_time_us":889438,"threads_started":1}]]}
W20251214 20:52:32.022661 989 leader_election.cc:343] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.022764 989 leader_election.cc:304] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.023499 2802 raft_consensus.cc:2749] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.024387 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 534) took 2124 ms. Trace:
I20251214 20:52:32.024483 1856 rpcz_store.cc:276] 1214 20:52:29.899767 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.903156 (+ 3389us) service_pool.cc:225] Handling call
1214 20:52:31.289358 (+1386202us) ts_tablet_manager.cc:1151] Deleting tablet b1f4ea72c6274569a98cab6afdc55d55
1214 20:52:31.289368 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:31.611003 (+321635us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:31.946667 (+335664us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.024363 (+ 77696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1385478}
I20251214 20:52:32.024839 1856 tablet_replica.cc:333] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.025022 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b7dd0ab78534a34ba342ec6e4fae129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.025074 1856 raft_consensus.cc:2243] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:32.025033 879 tablet_service.cc:1552] Processing DeleteTablet for tablet f84e9dfc23d34eb0a379e8ac5a442e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:15 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:32.025398 1856 raft_consensus.cc:2272] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.025956 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b68367f902f4be386788a8d138b5762 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.026007 1856 ts_tablet_manager.cc:1916] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.026700 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet b1f4ea72c6274569a98cab6afdc55d55 (table default.Table_893229542bec44e8b903bb0934bd16f5 [id=83b21f4baa574636ae3b50358fefae95]) successfully deleted
I20251214 20:52:32.027612 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 274185a4115b458e8e9a376bdbcc870b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.027741 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet fc7353593bda4fceaa389c393ddc19c6 with delete_type TABLET_DATA_DELETED (replaced by tablet a9c383e343544c83ada89bea6ab20e94) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.027777 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet feb19ea389af4ea6bb158a0e8aaeb094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.027947 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet e6ebfc373fbf41b1920c271a15302484 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.028069 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet b69bb4d5979f492683fc13238a668b9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.028167 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet bc73d590531c409e9f60670435dc8223 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.028883 1334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 5 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4"
I20251214 20:52:32.028975 446 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_ee7a207364d74e6b81d6a72e57cacbd5" } new_table_name: "default.Table_0d865be331094b9d8c407969618d12d4" modify_external_catalogs: true
I20251214 20:52:32.030596 452 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_7fbc660486544d9fb711dcdca0660237" } new_table_name: "default.Table_f9c22ae822b3401ab43ad74df0b0289c" modify_external_catalogs: true
I20251214 20:52:32.031337 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_31f83a8d780849d6b8b604d5223bd5a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:32.031550 447 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_581df19239f343f89fbe7e5c17d6d85c" } modify_external_catalogs: true
I20251214 20:52:32.031774 447 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50088:
table { table_name: "default.Table_581df19239f343f89fbe7e5c17d6d85c" } modify_external_catalogs: true
W20251214 20:52:32.031911 455 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31f83a8d780849d6b8b604d5223bd5a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:32.032379 449 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50088:
name: "default.Table_28893b766f8a4c93a8681c5e59752252"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:32.032970 1203 tablet_bootstrap.cc:492] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
W20251214 20:52:32.032982 449 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28893b766f8a4c93a8681c5e59752252 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:32.033113 1203 ts_tablet_manager.cc:1403] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.006s
I20251214 20:52:32.033388 2291 leader_election.cc:304] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6713a20a82b434f8a8bc122113e26ad; no voters: 7c48b8eaf30c4e2caa94a51d087406d4, c27224391721467a8f46e467a980b37a
I20251214 20:52:32.034300 1203 raft_consensus.cc:359] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.034452 1203 raft_consensus.cc:385] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.034494 1203 raft_consensus.cc:740] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.034693 1203 consensus_queue.cc:260] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.041227 1977 ts_tablet_manager.cc:1929] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251214 20:52:32.041388 1977 log.cc:1199] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/956e4b4493f24109ab1251dde0a4cfc0
I20251214 20:52:32.041770 1977 ts_tablet_manager.cc:1950] T 956e4b4493f24109ab1251dde0a4cfc0 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.041792 2688 raft_consensus.cc:3055] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 4 LEADER]: Stepping down as leader of term 4
I20251214 20:52:32.041911 2688 raft_consensus.cc:740] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c27224391721467a8f46e467a980b37a, State: Running, Role: LEADER
I20251214 20:52:32.042014 2816 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251214 20:52:32.043349 2688 consensus_queue.cc:260] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 955, Committed index: 955, Last appended: 4.955, Last appended by leader: 955, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:32.043712 2688 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 4 FOLLOWER]: Advancing to term 5
I20251214 20:52:32.044055 453 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet e9f6da5d29294f08a86c96c6821dda10
I20251214 20:52:32.044691 453 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 523) took 1094 ms. Trace:
I20251214 20:52:32.044818 453 rpcz_store.cc:276] 1214 20:52:30.950670 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.056752 (+106082us) service_pool.cc:225] Handling call
1214 20:52:31.070553 (+ 13801us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b1811b6 00007ff411c8c486 00007ff411c8c1b5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:31.071864 (+ 1311us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:31.072375 (+ 511us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:31.072450 (+ 75us) catalog_manager.cc:2824] Locking tablets
1214 20:52:31.110243 (+ 37793us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:32.043792 (+933549us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:32.043909 (+ 117us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:32.043967 (+ 58us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:32.044641 (+ 674us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:31.114820 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:31.114903 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:31.114908 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:31.114910 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:31.114988 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:31.114996 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:31.115000 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:31.115098 (+ 98us) tablet.cc:699] Row locks acquired
1214 20:52:31.115101 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:31.121100 (+ 5999us) write_op.cc:270] Start()
1214 20:52:31.121148 (+ 48us) write_op.cc:276] Timestamp: P: 1765745551115067 usec, L: 0
1214 20:52:31.121153 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:31.121381 (+ 228us) log.cc:844] Serialized 756 byte log entry
1214 20:52:32.038035 (+916654us) op_driver.cc:464] REPLICATION: finished
1214 20:52:32.042253 (+ 4218us) write_op.cc:301] APPLY: starting
1214 20:52:32.042279 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:32.042342 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:32.042398 (+ 56us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:32.042492 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:32.042535 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:32.042542 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:32.042660 (+ 118us) log.cc:844] Serialized 29 byte log entry
1214 20:52:32.042747 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:32.042776 (+ 29us) write_op.cc:454] Released schema lock
1214 20:52:32.042785 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:32.042867 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:32.042871 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":30999808,"child_traces":[["op",{"apply.queue_time_us":4133,"apply.run_cpu_time_us":617,"apply.run_wall_time_us":619,"num_ops":2,"prepare.queue_time_us":3896,"prepare.run_cpu_time_us":705,"prepare.run_wall_time_us":6621,"replication_time_us":916841}]]}
I20251214 20:52:32.046221 454 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a: Sending DeleteTablet for 3 replicas of tablet c07966670d294a0495ec54d48c7f09cd
W20251214 20:52:32.046386 450 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bd89709abbc49809a2cdb6d0c29fdb8 consensus state has no leader
I20251214 20:52:32.049124 879 tablet_service.cc:1552] Processing DeleteTablet for tablet c07966670d294a0495ec54d48c7f09cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:32.049715 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 410) took 4061 ms. Trace:
I20251214 20:52:32.049830 1982 rpcz_store.cc:276] 1214 20:52:27.987773 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:27.988353 (+ 580us) service_pool.cc:225] Handling call
1214 20:52:30.553453 (+2565100us) ts_tablet_manager.cc:1151] Deleting tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf
1214 20:52:30.553464 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:31.643405 (+1089941us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:31.947299 (+303894us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.049679 (+102380us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2564461}
I20251214 20:52:32.050235 1982 tablet_replica.cc:333] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.050415 746 tablet_service.cc:1552] Processing DeleteTablet for tablet c07966670d294a0495ec54d48c7f09cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:32.050535 753 tablet_service.cc:1552] Processing DeleteTablet for tablet e9f6da5d29294f08a86c96c6821dda10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:57406
I20251214 20:52:32.050662 894 tablet_service.cc:1552] Processing DeleteTablet for tablet e9f6da5d29294f08a86c96c6821dda10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:37530
I20251214 20:52:32.050921 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet a38e82f5dd1442f3a1dfcd6fbd94dbaf (table default.Table_08488285c9094a9dacc1e489c50271b3 [id=d3b4eef101ec491485c1889f08de9992]) successfully deleted
I20251214 20:52:32.051633 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet c07966670d294a0495ec54d48c7f09cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.051726 1203 ts_tablet_manager.cc:1434] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251214 20:52:32.052745 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 430) took 3680 ms. Trace:
I20251214 20:52:32.052855 1977 rpcz_store.cc:276] 1214 20:52:28.371836 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.371887 (+ 51us) service_pool.cc:225] Handling call
1214 20:52:31.914291 (+3542404us) ts_tablet_manager.cc:1151] Deleting tablet 956e4b4493f24109ab1251dde0a4cfc0
1214 20:52:31.914300 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:31.940887 (+ 26587us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.041047 (+100160us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.052718 (+ 11671us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3542166}
I20251214 20:52:32.053365 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]) successfully deleted
I20251214 20:52:32.057281 1982 raft_consensus.cc:2243] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:32.057694 1977 tablet_replica.cc:333] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.057922 1977 raft_consensus.cc:2243] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.058046 1856 ts_tablet_manager.cc:1929] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:32.058104 1977 raft_consensus.cc:2272] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.058171 1856 log.cc:1199] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/bf37c50e511f454ca4887c7f6ca1a746
I20251214 20:52:32.058499 1856 ts_tablet_manager.cc:1950] T bf37c50e511f454ca4887c7f6ca1a746 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.058676 2787 raft_consensus.cc:493] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.058723 1977 ts_tablet_manager.cc:1916] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.058838 2787 raft_consensus.cc:515] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.059079 1982 raft_consensus.cc:2272] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.059478 2787 leader_election.cc:290] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:32.059795 1982 ts_tablet_manager.cc:1916] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.060369 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1835374cfa48939d72476870accc3b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.060964 848 leader_election.cc:343] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:32.063201 851 leader_election.cc:343] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 9e1835374cfa48939d72476870accc3b
I20251214 20:52:32.063347 851 leader_election.cc:304] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:32.063755 2787 raft_consensus.cc:2749] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.065788 1203 tablet_bootstrap.cc:492] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.067732 1203 tablet_bootstrap.cc:654] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.068812 1202 tablet_bootstrap.cc:492] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.068965 1202 ts_tablet_manager.cc:1403] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.088s user 0.005s sys 0.005s
I20251214 20:52:32.069823 1202 raft_consensus.cc:359] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.069983 1202 raft_consensus.cc:385] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.070030 1202 raft_consensus.cc:740] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.070250 1202 consensus_queue.cc:260] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.070657 453 master_service.cc:954] ReplaceTablet: received request to replace tablet c7248a2e8355425fa051a197160dbdbb from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:32.070870 454 master_service.cc:954] ReplaceTablet: received request to replace tablet 0c62ffb71fff43d4812eb86f40ce1c58 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:32.071171 453 master_service.cc:954] ReplaceTablet: received request to replace tablet 06ebe9f717bf43108748605cf4f961b7 from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:32.071554 453 master_service.cc:954] ReplaceTablet: received request to replace tablet fa141b2570514c6aacb2850e4bfd1869 from {username='slave'} at 127.0.0.1:50088
W20251214 20:52:32.074062 1111 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.62:42865)
W20251214 20:52:32.074368 1111 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:42865)
W20251214 20:52:32.075083 1111 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:42865)
I20251214 20:52:32.084091 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 548) took 2115 ms. Trace:
I20251214 20:52:32.084267 1856 rpcz_store.cc:276] 1214 20:52:29.969021 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.969075 (+ 54us) service_pool.cc:225] Handling call
1214 20:52:32.024786 (+2055711us) ts_tablet_manager.cc:1151] Deleting tablet bf37c50e511f454ca4887c7f6ca1a746
1214 20:52:32.024796 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.054835 (+ 30039us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.057932 (+ 3097us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.084043 (+ 26111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2055491}
I20251214 20:52:32.084757 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 564) took 1882 ms. Trace:
I20251214 20:52:32.084797 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet bf37c50e511f454ca4887c7f6ca1a746 (table default.Table_af66f4d0779b46a5a91d7d2490a845c9 [id=4616162e7f9c4c55a4dd98c375d2c2e9]) successfully deleted
I20251214 20:52:32.084839 1856 rpcz_store.cc:276] 1214 20:52:30.201866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.202033 (+ 167us) service_pool.cc:225] Handling call
1214 20:52:32.084594 (+1882561us) ts_tablet_manager.cc:1151] Deleting tablet a6df388bdfbc40c183c4b14af2336a6a
1214 20:52:32.084605 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.084741 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1882307}
I20251214 20:52:32.085098 1856 tablet_replica.cc:333] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.085292 1856 raft_consensus.cc:2243] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
W20251214 20:52:32.085470 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet a6df388bdfbc40c183c4b14af2336a6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6df388bdfbc40c183c4b14af2336a6a already in progress: creating tablet
I20251214 20:52:32.085623 1856 raft_consensus.cc:2272] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.086256 1856 ts_tablet_manager.cc:1916] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.087548 1202 ts_tablet_manager.cc:1434] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251214 20:52:32.087949 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet e9f6da5d29294f08a86c96c6821dda10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.091284 2777 raft_consensus.cc:515] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:32.093822 851 leader_election.cc:343] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 98b7cac3930a4a9eb16ac413c8f1e092
I20251214 20:52:32.094342 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 37d198fa8da642dca442476578a35f1e (DEFAULT_TABLE table=default.Table_81b328890d6748ea8a95ac3cd696c9dd [id=4a964e807c8a4161bf9fa9cb732ba21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:32.095022 1202 tablet_bootstrap.cc:492] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.095191 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98b7cac3930a4a9eb16ac413c8f1e092" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:32.095393 768 raft_consensus.cc:3060] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 2
I20251214 20:52:32.096760 1202 tablet_bootstrap.cc:654] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.096899 2777 leader_election.cc:290] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:32.100441 2787 raft_consensus.cc:493] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.101564 2787 raft_consensus.cc:515] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.102200 2787 leader_election.cc:290] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.102963 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d424e3387698434f92e84eb1d43f5949" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.103900 848 leader_election.cc:343] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.105966 453 master_service.cc:954] ReplaceTablet: received request to replace tablet b8727f302d35471eb2eac1e77b8d966b from {username='slave'} at 127.0.0.1:50088
I20251214 20:52:32.106087 1856 ts_tablet_manager.cc:1929] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.109653 768 raft_consensus.cc:2468] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 2.
I20251214 20:52:32.110114 848 leader_election.cc:304] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:32.110260 455 catalog_manager.cc:2202] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.068s ago.
I20251214 20:52:32.110527 2777 raft_consensus.cc:2804] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 2 FOLLOWER]: Leader election won for term 2
I20251214 20:52:32.111106 2777 raft_consensus.cc:697] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 2 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:32.111251 1982 ts_tablet_manager.cc:1929] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:32.111402 2777 consensus_queue.cc:237] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.111663 1982 log.cc:1199] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/d13f93bb5e5f4b59bcf9f4bfed070f25
I20251214 20:52:32.112039 1982 ts_tablet_manager.cc:1950] T d13f93bb5e5f4b59bcf9f4bfed070f25 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.113267 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 413) took 3822 ms. Trace:
I20251214 20:52:32.113440 1982 rpcz_store.cc:276] 1214 20:52:28.290527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.290600 (+ 73us) service_pool.cc:225] Handling call
1214 20:52:32.050173 (+3759573us) ts_tablet_manager.cc:1151] Deleting tablet d13f93bb5e5f4b59bcf9f4bfed070f25
1214 20:52:32.050185 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.108258 (+ 58073us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.111119 (+ 2861us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.113242 (+ 2123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3759286}
I20251214 20:52:32.114105 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet d13f93bb5e5f4b59bcf9f4bfed070f25 (table default.Table_b971804db7ad428a88f22d83a1de29d9 [id=a8e124ed5b7141318dfa07874e188d10]) successfully deleted
I20251214 20:52:32.115499 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 414) took 3824 ms. Trace:
I20251214 20:52:32.115638 1982 rpcz_store.cc:276] 1214 20:52:28.290623 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.290734 (+ 111us) service_pool.cc:225] Handling call
1214 20:52:32.115330 (+3824596us) ts_tablet_manager.cc:1151] Deleting tablet a0976ffcc649461fac74e20b5e402275
1214 20:52:32.115344 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.115479 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3824375}
W20251214 20:52:32.115778 446 catalog_manager.cc:3922] Illegal state: an error occurred while updating the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.067s ago.
W20251214 20:52:32.116294 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet a0976ffcc649461fac74e20b5e402275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0976ffcc649461fac74e20b5e402275 already in progress: creating tablet
I20251214 20:52:32.116761 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 415) took 3825 ms. Trace:
I20251214 20:52:32.116882 1982 rpcz_store.cc:276] 1214 20:52:28.290890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.290957 (+ 67us) service_pool.cc:225] Handling call
1214 20:52:32.116639 (+3825682us) ts_tablet_manager.cc:1151] Deleting tablet 0a2fbec0543445b7856ef22c934c6840
1214 20:52:32.116651 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.116744 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3825540}
W20251214 20:52:32.117370 452 catalog_manager.cc:3922] Illegal state: an error occurred while updating the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.068s ago.
W20251214 20:52:32.117372 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 0a2fbec0543445b7856ef22c934c6840 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2fbec0543445b7856ef22c934c6840 already in progress: creating tablet
I20251214 20:52:32.117810 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 416) took 3826 ms. Trace:
I20251214 20:52:32.117914 1982 rpcz_store.cc:276] 1214 20:52:28.290995 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.291040 (+ 45us) service_pool.cc:225] Handling call
1214 20:52:32.117708 (+3826668us) ts_tablet_manager.cc:1151] Deleting tablet 088c72a6f5014b6fbc6ee060b6b7a590
1214 20:52:32.117717 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.117792 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3826527}
W20251214 20:52:32.118386 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 088c72a6f5014b6fbc6ee060b6b7a590 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 088c72a6f5014b6fbc6ee060b6b7a590 already in progress: creating tablet
I20251214 20:52:32.118814 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 412) took 3828 ms. Trace:
W20251214 20:52:32.119962 449 catalog_manager.cc:2202] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.077s ago.
I20251214 20:52:32.121567 1977 ts_tablet_manager.cc:1929] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.121735 1977 log.cc:1199] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/69f6108792fa4c2a9744e3248c76251f
I20251214 20:52:32.121367 2819 raft_consensus.cc:493] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.121977 1982 rpcz_store.cc:276] 1214 20:52:28.290432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.295749 (+ 5317us) service_pool.cc:225] Handling call
1214 20:52:32.118712 (+3822963us) ts_tablet_manager.cc:1151] Deleting tablet 453cf734f8b34b59aa246d618ffeade5
1214 20:52:32.118722 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.118797 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3822713}
I20251214 20:52:32.122063 2819 raft_consensus.cc:515] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:32.122496 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 453cf734f8b34b59aa246d618ffeade5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453cf734f8b34b59aa246d618ffeade5 already in progress: creating tablet
I20251214 20:52:32.122679 2819 leader_election.cc:290] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.122808 1982 tablet_replica.cc:333] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.123106 1982 raft_consensus.cc:2243] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.123330 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9b1a937150ce4ecfa210c8465d682496" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.123571 910 raft_consensus.cc:2393] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate 63848a9fb78a4013a6f1657621272d75 in this term.
I20251214 20:52:32.123739 1977 ts_tablet_manager.cc:1950] T 69f6108792fa4c2a9744e3248c76251f P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
W20251214 20:52:32.123781 713 leader_election.cc:343] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 9b1a937150ce4ecfa210c8465d682496
I20251214 20:52:32.123987 712 leader_election.cc:304] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.124410 2819 raft_consensus.cc:3060] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.124614 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 431) took 3727 ms. Trace:
I20251214 20:52:32.124696 1977 rpcz_store.cc:276] 1214 20:52:28.397314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.397374 (+ 60us) service_pool.cc:225] Handling call
1214 20:52:32.057613 (+3660239us) ts_tablet_manager.cc:1151] Deleting tablet 69f6108792fa4c2a9744e3248c76251f
1214 20:52:32.057629 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.064251 (+ 6622us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.121372 (+ 57121us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.124591 (+ 3219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3660012}
I20251214 20:52:32.125029 1977 tablet_replica.cc:333] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.125228 1977 raft_consensus.cc:2243] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:32.125531 1977 raft_consensus.cc:2272] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.126084 1977 ts_tablet_manager.cc:1916] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.127187 1982 raft_consensus.cc:2272] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.127521 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 69f6108792fa4c2a9744e3248c76251f (table default.Table_b3f5ab40d9334de1a194aab3347387d7 [id=b470f1b24a6d4e7ab4420ae212e54ff0]) successfully deleted
I20251214 20:52:32.127766 2819 raft_consensus.cc:2749] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.129241 1982 ts_tablet_manager.cc:1916] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:32.131697 447 catalog_manager.cc:2848] Illegal state: an error occurred while updating the sys-catalog: replica c27224391721467a8f46e467a980b37a is not leader of this config: current role FOLLOWER
I20251214 20:52:32.132004 447 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50088 (request call id 530) took 1015 ms. Trace:
I20251214 20:52:32.132143 447 rpcz_store.cc:276] 1214 20:52:31.116703 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.976293 (+859590us) service_pool.cc:225] Handling call
1214 20:52:32.031449 (+ 55156us) spinlock_profiling.cc:243] Waited 54.8 ms on lock 0x616000031354. stack: 00007ff407447ee6 00007ff407447ca0 00007ff4064233ad 0000000000984f94 0000000000984f5d 00007ff41676d6bd 00007ff41676d7b3 00007ff413334521 00007ff41aff396d 00007ff41b1811b6 00007ff411c8c486 00007ff411c8c1b5 00007ff40e40656e 00007ff40e40544e 00007ff40e40b685 00007ff40e40bfaf
1214 20:52:32.031864 (+ 415us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:32.032021 (+ 157us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:32.032107 (+ 86us) catalog_manager.cc:2824] Locking tablets
1214 20:52:32.130585 (+ 98478us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:32.131532 (+ 947us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:32.131535 (+ 3us) write_op.cc:454] Released schema lock
1214 20:52:32.131958 (+ 423us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:32.131452 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica c27224391721467a8f46e467a980b37a is not leader of this config: current role FOLLOWER)
1214 20:52:32.131466 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:32.131475 (+ 9us) write_op.cc:454] Released schema lock
1214 20:52:32.131481 (+ 6us) write_op.cc:335] FINISH: Op aborted
Metrics: {"spinlock_wait_cycles":125977344,"child_traces":[["op",{}]]}
W20251214 20:52:32.133512 1111 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:42865)
I20251214 20:52:32.135712 1202 tablet_bootstrap.cc:492] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.135993 1202 ts_tablet_manager.cc:1403] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.001s
I20251214 20:52:32.136842 1202 raft_consensus.cc:359] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.137111 1202 raft_consensus.cc:385] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.137202 1202 raft_consensus.cc:740] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.137449 1202 consensus_queue.cc:260] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.138180 1202 ts_tablet_manager.cc:1434] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:32.138499 1202 tablet_bootstrap.cc:492] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.101450 1203 tablet_bootstrap.cc:492] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.139256 1856 log.cc:1199] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/96dc4bdabb234c3fa4c26bd1633e783d
I20251214 20:52:32.139328 1203 ts_tablet_manager.cc:1403] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20251214 20:52:32.139690 1856 ts_tablet_manager.cc:1950] T 96dc4bdabb234c3fa4c26bd1633e783d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.139562 448 catalog_manager.cc:5654] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:32.140022 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 3a23561b9f4e48bbbd796e14fafec086 (table default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3 [id=f1338e81f4ec4dce87130f2ddbe98041]). Expected version 1 got 0
I20251214 20:52:32.140112 1203 raft_consensus.cc:359] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.140273 1203 raft_consensus.cc:385] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.140321 1203 raft_consensus.cc:740] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.140369 448 catalog_manager.cc:5654] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } }
I20251214 20:52:32.140532 1203 consensus_queue.cc:260] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.140729 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_5de6830aaec645f2992e4cfaceb20c27 [id=9553b46856c94003a293135c814a7df7]). Expected version 2 got 0
I20251214 20:52:32.104415 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d424e3387698434f92e84eb1d43f5949" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:32.140944 1050 raft_consensus.cc:2468] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:32.140959 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (table default.Table_fcafcf3c6ddb4457aea41320f7bf0b74 [id=4329d01e7f6147d0adad53fa86369b17]). Expected version 1 got 0
I20251214 20:52:32.098990 2735 raft_consensus.cc:493] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.141208 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_9060872cac6842548b9f1f80d16dbe19 [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 2 got 0
I20251214 20:52:32.141215 2735 raft_consensus.cc:515] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.141414 448 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d424e3387698434f92e84eb1d43f5949 (table default.Table_9098a11909c844dca6d4e80a284aa2e8 [id=1597e24e3fff4007ab66b44505c960d8]). Expected version 1 got 0
I20251214 20:52:32.141821 2735 leader_election.cc:290] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
E20251214 20:52:32.142272 448 catalog_manager.cc:5809] Error updating tablets from a2e8fd1cb731446d88f8e510c7a52999: Illegal state: replica c27224391721467a8f46e467a980b37a is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "876d2a1f1d1c4f6882a5857b43568ee5" state: RUNNING consensus_state { current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "b69bb4d5979f492683fc13238a668b9f" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "57f2314d33fa474ca36dafe0a0eae5ae" state: RUNNING consensus_state { current_term: 2 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "c6a8863ace4541aea7463af24db27142" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "b1f4ea72c6274569a98cab6afdc55d55" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d424e3387698434f92e84eb1d43f5949" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "9e1835374cfa48939d72476870accc3b" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "719d157ac8f74eb7911df05ab5e4a3d5" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e6bdfbe4d15345de8be5de00bfe4ca07" state: RUNNING consensus_state { current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "492770a7b8b74ba49315f974cc54eedb" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "738668adf2874d6894db6fa875f6990c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27319 live_row_count: 0 } } updated_tablets { tablet_id: "3a23561b9f4e48bbbd796e14fafec086" state: RUNNING consensus_state { current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } sequence_number: 201
I20251214 20:52:32.143321 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 565) took 1941 ms. Trace:
I20251214 20:52:32.143410 1856 rpcz_store.cc:276] 1214 20:52:30.201979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.206082 (+ 4103us) service_pool.cc:225] Handling call
1214 20:52:32.085047 (+1878965us) ts_tablet_manager.cc:1151] Deleting tablet 96dc4bdabb234c3fa4c26bd1633e783d
1214 20:52:32.085054 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.102796 (+ 17742us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.105977 (+ 3181us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.143300 (+ 37323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1878756}
I20251214 20:52:32.145233 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0976ffcc649461fac74e20b5e402275" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.146466 1206 tablet_bootstrap.cc:492] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.146588 1206 ts_tablet_manager.cc:1403] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.170s user 0.007s sys 0.005s
I20251214 20:52:32.147269 1206 raft_consensus.cc:359] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.147408 1206 raft_consensus.cc:385] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.147449 1206 raft_consensus.cc:740] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.147626 1206 consensus_queue.cc:260] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.148087 1206 ts_tablet_manager.cc:1434] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251214 20:52:32.149075 1977 ts_tablet_manager.cc:1929] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251214 20:52:32.149194 1977 log.cc:1199] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/b70579ddf96f4499b4e4f235e8f69d67
I20251214 20:52:32.149530 1977 ts_tablet_manager.cc:1950] T b70579ddf96f4499b4e4f235e8f69d67 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.149621 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0976ffcc649461fac74e20b5e402275" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.150154 851 leader_election.cc:304] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.150259 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 96dc4bdabb234c3fa4c26bd1633e783d (table default.Table_5e8a4451754247c2add9a3245b63b050 [id=d7f40373ddad4286a8fe7966df2d9668]) successfully deleted
I20251214 20:52:32.150352 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 432) took 3736 ms. Trace:
W20251214 20:52:32.150403 989 leader_election.cc:343] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.150544 2787 raft_consensus.cc:2804] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:32.150638 2787 raft_consensus.cc:493] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:32.150693 2787 raft_consensus.cc:3060] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.150713 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 566) took 1941 ms. Trace:
I20251214 20:52:32.150794 1856 rpcz_store.cc:276] 1214 20:52:30.208781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.208856 (+ 75us) service_pool.cc:225] Handling call
1214 20:52:32.143664 (+1934808us) ts_tablet_manager.cc:1151] Deleting tablet 44c61824bacd406280ba03357f9f5b1d
1214 20:52:32.143673 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.150692 (+ 7019us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1934630}
W20251214 20:52:32.151337 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 44c61824bacd406280ba03357f9f5b1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44c61824bacd406280ba03357f9f5b1d already in progress: creating tablet
I20251214 20:52:32.151764 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 563) took 1957 ms. Trace:
I20251214 20:52:32.151933 1856 rpcz_store.cc:276] 1214 20:52:30.193978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.214128 (+ 20150us) service_pool.cc:225] Handling call
1214 20:52:32.151646 (+1937518us) ts_tablet_manager.cc:1151] Deleting tablet c249c096a41f4b4093eaf9afc6361249
1214 20:52:32.151659 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.151748 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1937207}
W20251214 20:52:32.152567 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c249c096a41f4b4093eaf9afc6361249 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c249c096a41f4b4093eaf9afc6361249 already in progress: creating tablet
I20251214 20:52:32.153191 2787 raft_consensus.cc:515] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.153729 2787 leader_election.cc:290] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.154081 1206 tablet_bootstrap.cc:492] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.154958 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d424e3387698434f92e84eb1d43f5949" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:32.155448 1206 tablet_bootstrap.cc:654] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:32.155467 848 leader_election.cc:343] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.155752 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 562) took 1974 ms. Trace:
I20251214 20:52:32.155872 1856 rpcz_store.cc:276] 1214 20:52:30.181097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.220013 (+ 38916us) service_pool.cc:225] Handling call
1214 20:52:32.155050 (+1935037us) ts_tablet_manager.cc:1151] Deleting tablet 18e95ff0e0da4c7680f594ba31cdc3a1
1214 20:52:32.155062 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.155310 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1934699}
I20251214 20:52:32.103480 2802 raft_consensus.cc:493] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:32.156271 991 leader_election.cc:343] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.156291 2802 raft_consensus.cc:515] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.156018 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d424e3387698434f92e84eb1d43f5949" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
W20251214 20:52:32.156509 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 18e95ff0e0da4c7680f594ba31cdc3a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18e95ff0e0da4c7680f594ba31cdc3a1 already in progress: creating tablet
I20251214 20:52:32.156545 1050 raft_consensus.cc:3060] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.156821 1856 tablet_replica.cc:333] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.156957 2802 leader_election.cc:290] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.157047 1856 raft_consensus.cc:2243] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.157032 451 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d424e3387698434f92e84eb1d43f5949 (table default.Table_9098a11909c844dca6d4e80a284aa2e8 [id=1597e24e3fff4007ab66b44505c960d8]). Expected version 1 got 0
I20251214 20:52:32.157207 1856 raft_consensus.cc:2272] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.157478 451 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (table default.Table_fcafcf3c6ddb4457aea41320f7bf0b74 [id=4329d01e7f6147d0adad53fa86369b17]). Expected version 1 got 0
I20251214 20:52:32.157640 451 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet f11834cabdad445b8e53cd237d146282 (table default.Table_398f36b3acb440d88213e38f63740ca5 [id=954b3b4acdd94043b2608438d00483e7]). Expected version 1 got 0
I20251214 20:52:32.157783 1856 ts_tablet_manager.cc:1916] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.157811 451 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 082ec70b8ee646b898d9201ad073d292 (table default.Table_e3feb6d6cacb48dca8455eacf4cfb84b [id=8f720514e13b45689817a9811fb624d9]). Expected version 1 got 0
I20251214 20:52:32.158032 451 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:32.156386 991 leader_election.cc:304] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.159713 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f945814952b14131ba4e0c67a403a680" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.160084 2802 raft_consensus.cc:2749] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:32.160344 989 leader_election.cc:343] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.160431 1982 ts_tablet_manager.cc:1929] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251214 20:52:32.160544 1982 log.cc:1199] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/956e4b4493f24109ab1251dde0a4cfc0
I20251214 20:52:32.160916 1982 ts_tablet_manager.cc:1950] T 956e4b4493f24109ab1251dde0a4cfc0 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.161075 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f945814952b14131ba4e0c67a403a680" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:32.162461 991 leader_election.cc:343] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.162572 991 leader_election.cc:304] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.162699 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 96dc4bdabb234c3fa4c26bd1633e783d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.162883 2802 raft_consensus.cc:2749] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.165704 1050 raft_consensus.cc:2468] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:32.166482 851 leader_election.cc:304] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.167227 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 417) took 3783 ms. Trace:
I20251214 20:52:32.167385 1982 rpcz_store.cc:276] 1214 20:52:28.384111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.384171 (+ 60us) service_pool.cc:225] Handling call
1214 20:52:32.122751 (+3738580us) ts_tablet_manager.cc:1151] Deleting tablet 956e4b4493f24109ab1251dde0a4cfc0
1214 20:52:32.122763 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.142207 (+ 19444us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.160291 (+ 18084us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.167171 (+ 6880us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3738270}
I20251214 20:52:32.168078 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 956e4b4493f24109ab1251dde0a4cfc0 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]) successfully deleted
I20251214 20:52:32.150439 1977 rpcz_store.cc:276] 1214 20:52:28.413955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.414010 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:32.124980 (+3710970us) ts_tablet_manager.cc:1151] Deleting tablet b70579ddf96f4499b4e4f235e8f69d67
1214 20:52:32.124989 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.145983 (+ 20994us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.148958 (+ 2975us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.150332 (+ 1374us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3710793}
I20251214 20:52:32.168825 1982 tablet_replica.cc:333] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.169169 1977 tablet_replica.cc:333] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.169589 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_3a068835084e406eb4047a827faeb5d7 [id=bdb017130eed4d76b635603560c91d41]) successfully deleted
I20251214 20:52:32.169620 1982 raft_consensus.cc:2243] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.169948 1977 raft_consensus.cc:2243] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.170414 1982 raft_consensus.cc:2272] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.170495 1977 raft_consensus.cc:2272] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.106520 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d198fa8da642dca442476578a35f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:32.171296 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet f1eb4c3aa654480caaa69dbd5047365c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.171962 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet a0976ffcc649461fac74e20b5e402275 with delete_type TABLET_DATA_DELETED (replaced by tablet 526f28f9aaad445081d9d69cc20ed7b3) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.174299 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet ae948ffc81764692b92045f5befe0184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.174811 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet eb29fd8c85344ed8aa8d880e2ed240bc with delete_type TABLET_DATA_DELETED (replaced by tablet 7242be189ca54c9182181e33d6743e6b) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.174924 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24e4f4e74040249068505058c15622 with delete_type TABLET_DATA_DELETED (replaced by tablet 0dbfe8ee68bf4a2c88489b1f81b36991) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.175470 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 088c72a6f5014b6fbc6ee060b6b7a590 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.175510 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 3b47ca1a03944cf1847d3578856dafa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.176064 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet d01db63fd5d2474a9defdce0b98fc29d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.176245 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d7c7b3f62764ff497f4d70b53eed7de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:20 UTC) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.176499 2787 raft_consensus.cc:2804] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:32.176641 2787 raft_consensus.cc:697] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:32.176877 2787 consensus_queue.cc:237] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
E20251214 20:52:32.177440 666 catalog_manager.cc:826] Error processing pending assignments: Illegal state: error persisting updated tablet metadata: replica c27224391721467a8f46e467a980b37a is not leader of this config: current role FOLLOWER
I20251214 20:52:32.177961 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 65e79db74fa046698507c83416ee1669 with delete_type TABLET_DATA_DELETED (replaced by tablet 4dcd511a5dca4a2cbda81fbc255ce354) from {username='slave'} at 127.0.0.1:40270
I20251214 20:52:32.179651 1982 ts_tablet_manager.cc:1916] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.187376 1977 ts_tablet_manager.cc:1916] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.190537 1202 tablet_bootstrap.cc:654] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.191978 1856 ts_tablet_manager.cc:1929] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.192138 1856 log.cc:1199] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/6a000951f61948ac85fdd431f3d162c2
I20251214 20:52:32.192451 1856 ts_tablet_manager.cc:1950] T 6a000951f61948ac85fdd431f3d162c2 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.193252 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 567) took 1915 ms. Trace:
I20251214 20:52:32.193341 1856 rpcz_store.cc:276] 1214 20:52:30.277775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.280226 (+ 2451us) service_pool.cc:225] Handling call
1214 20:52:32.156772 (+1876546us) ts_tablet_manager.cc:1151] Deleting tablet 6a000951f61948ac85fdd431f3d162c2
1214 20:52:32.156781 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.175877 (+ 19096us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.179098 (+ 3221us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.193229 (+ 14131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1876354}
I20251214 20:52:32.193774 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 568) took 1761 ms. Trace:
I20251214 20:52:32.193854 1856 rpcz_store.cc:276] 1214 20:52:30.432740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.433179 (+ 439us) service_pool.cc:225] Handling call
1214 20:52:32.193638 (+1760459us) ts_tablet_manager.cc:1151] Deleting tablet cbefd8f111084785bafd7e74ac443fd0
1214 20:52:32.193648 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.193759 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1760270}
I20251214 20:52:32.193871 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 6a000951f61948ac85fdd431f3d162c2 (table default.Table_c9b7a105aebf4ed48e4162d811977686 [id=10887569aec34e4e91977b0d321c6904]) successfully deleted
I20251214 20:52:32.194146 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 569) took 1693 ms. Trace:
I20251214 20:52:32.194216 1856 rpcz_store.cc:276] 1214 20:52:30.501209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.501484 (+ 275us) service_pool.cc:225] Handling call
1214 20:52:32.194062 (+1692578us) ts_tablet_manager.cc:1151] Deleting tablet bd1161eebe884fe6b347bf517269c047
1214 20:52:32.194069 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.194134 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692332}
W20251214 20:52:32.194331 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet cbefd8f111084785bafd7e74ac443fd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbefd8f111084785bafd7e74ac443fd0 already in progress: creating tablet
I20251214 20:52:32.194438 1856 tablet_replica.cc:333] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.194656 1856 raft_consensus.cc:2243] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Raft consensus shutting down.
W20251214 20:52:32.194758 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet bd1161eebe884fe6b347bf517269c047 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd1161eebe884fe6b347bf517269c047 already in progress: creating tablet
I20251214 20:52:32.202654 2819 raft_consensus.cc:493] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.202812 2819 raft_consensus.cc:515] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.204493 2819 leader_election.cc:290] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:32.205955 712 leader_election.cc:343] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 47e813373fbd482298312ead4f037c01
I20251214 20:52:32.205996 1977 ts_tablet_manager.cc:1929] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.206105 1977 log.cc:1199] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/9b1a937150ce4ecfa210c8465d682496
I20251214 20:52:32.206396 1977 ts_tablet_manager.cc:1950] T 9b1a937150ce4ecfa210c8465d682496 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
W20251214 20:52:32.206709 713 leader_election.cc:343] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 47e813373fbd482298312ead4f037c01
I20251214 20:52:32.206813 713 leader_election.cc:304] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.207278 2819 raft_consensus.cc:2749] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.207295 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 433) took 3398 ms. Trace:
I20251214 20:52:32.207372 1977 rpcz_store.cc:276] 1214 20:52:28.809018 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.809279 (+ 261us) service_pool.cc:225] Handling call
1214 20:52:32.169005 (+3359726us) ts_tablet_manager.cc:1151] Deleting tablet 9b1a937150ce4ecfa210c8465d682496
1214 20:52:32.169022 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.196391 (+ 27369us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.205846 (+ 9455us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.207274 (+ 1428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3359393}
I20251214 20:52:32.207710 1977 tablet_replica.cc:333] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.207906 1977 raft_consensus.cc:2243] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 2 LEADER]: Raft consensus shutting down.
I20251214 20:52:32.208153 1977 pending_rounds.cc:70] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:32.208209 1977 pending_rounds.cc:77] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7232493779932942336 op_type: NO_OP noop_request { }
I20251214 20:52:32.208320 1977 raft_consensus.cc:2889] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:32.208408 1977 raft_consensus.cc:2272] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.208559 1202 tablet_bootstrap.cc:492] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.208710 1202 ts_tablet_manager.cc:1403] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.000s
I20251214 20:52:32.209447 1202 raft_consensus.cc:359] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.209841 1202 raft_consensus.cc:385] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.210036 1202 raft_consensus.cc:740] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.203804 1203 ts_tablet_manager.cc:1434] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.064s user 0.003s sys 0.007s
I20251214 20:52:32.210551 1203 tablet_bootstrap.cc:492] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.212492 1203 tablet_bootstrap.cc:654] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.213943 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 9b1a937150ce4ecfa210c8465d682496 (table default.Table_89dffab0bf1d4856b671651a07b3439e [id=822a24c7456947ffbbba51a79bac4f34]) successfully deleted
I20251214 20:52:32.214114 1856 raft_consensus.cc:2272] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.214741 1856 ts_tablet_manager.cc:1916] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.217047 1982 ts_tablet_manager.cc:1929] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.217170 1982 log.cc:1199] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/ec85857f047e44d78835305ddcb70768
I20251214 20:52:32.217501 1982 ts_tablet_manager.cc:1950] T ec85857f047e44d78835305ddcb70768 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.218358 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 419) took 3787 ms. Trace:
I20251214 20:52:32.218446 1982 rpcz_store.cc:276] 1214 20:52:28.430686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.430870 (+ 184us) service_pool.cc:225] Handling call
1214 20:52:32.168553 (+3737683us) ts_tablet_manager.cc:1151] Deleting tablet ec85857f047e44d78835305ddcb70768
1214 20:52:32.168670 (+ 117us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.182794 (+ 14124us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.216885 (+ 34091us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.218336 (+ 1451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3737426}
I20251214 20:52:32.218786 1982 tablet_replica.cc:333] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.219342 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet ec85857f047e44d78835305ddcb70768 (table default.Table_58a7d5bd289f408da592c143483b0329 [id=bb060058712440c7b1e5c40332a0263d]) successfully deleted
I20251214 20:52:32.223453 1202 consensus_queue.cc:260] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.227324 1203 tablet_bootstrap.cc:492] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.227488 1203 ts_tablet_manager.cc:1403] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20251214 20:52:32.228214 1203 raft_consensus.cc:359] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.228370 1203 raft_consensus.cc:385] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.228415 1203 raft_consensus.cc:740] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.228828 1203 consensus_queue.cc:260] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.229311 1203 ts_tablet_manager.cc:1434] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:32.229619 1203 tablet_bootstrap.cc:492] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.232613 1203 tablet_bootstrap.cc:654] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.218977 1982 raft_consensus.cc:2243] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:32.234004 1982 pending_rounds.cc:70] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: Trying to abort 1 pending ops.
I20251214 20:52:32.234064 1982 pending_rounds.cc:77] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493767851372544 op_type: NO_OP noop_request { }
I20251214 20:52:32.234163 1982 raft_consensus.cc:2889] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:32.234251 1982 raft_consensus.cc:2272] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.235651 1977 ts_tablet_manager.cc:1916] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.237376 2787 raft_consensus.cc:493] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.237565 2787 raft_consensus.cc:515] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.238183 2787 leader_election.cc:290] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:32.238960 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6a8863ace4541aea7463af24db27142" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.239343 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6a8863ace4541aea7463af24db27142" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:32.239492 848 leader_election.cc:343] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.239583 1050 raft_consensus.cc:2468] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:32.240065 851 leader_election.cc:304] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.240553 2787 raft_consensus.cc:2804] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:32.240619 2787 raft_consensus.cc:493] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:32.240670 2787 raft_consensus.cc:3060] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.242053 1856 ts_tablet_manager.cc:1929] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.242163 1856 log.cc:1199] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/e90d11d69b8d4fd49150aa67a8190e5b
I20251214 20:52:32.242480 1856 ts_tablet_manager.cc:1950] T e90d11d69b8d4fd49150aa67a8190e5b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.243321 2787 raft_consensus.cc:515] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.243472 1206 tablet_bootstrap.cc:492] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.243731 1206 ts_tablet_manager.cc:1403] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.000s
I20251214 20:52:32.244525 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6a8863ace4541aea7463af24db27142" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:32.244704 1050 raft_consensus.cc:3060] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.245055 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6a8863ace4541aea7463af24db27142" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:32.245568 848 leader_election.cc:343] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.243564 1982 ts_tablet_manager.cc:1916] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.247417 2787 leader_election.cc:290] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:32.247947 1050 raft_consensus.cc:2468] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:32.248512 851 leader_election.cc:304] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.248441 1206 raft_consensus.cc:359] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.248631 1206 raft_consensus.cc:385] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.248699 1206 raft_consensus.cc:740] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.248915 1206 consensus_queue.cc:260] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.249152 2787 raft_consensus.cc:2804] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:32.249778 1206 ts_tablet_manager.cc:1434] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251214 20:52:32.252977 2787 raft_consensus.cc:697] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:32.253221 1206 tablet_bootstrap.cc:492] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.253400 2787 consensus_queue.cc:237] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.255111 1206 tablet_bootstrap.cc:654] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.271279 1202 ts_tablet_manager.cc:1434] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
I20251214 20:52:32.271852 1202 tablet_bootstrap.cc:492] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.272814 1982 ts_tablet_manager.cc:1929] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.272957 1982 log.cc:1199] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/bf8643c7e4094f529cd55656c7b3edb6
I20251214 20:52:32.273299 1982 ts_tablet_manager.cc:1950] T bf8643c7e4094f529cd55656c7b3edb6 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.273507 1202 tablet_bootstrap.cc:654] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.276439 1206 tablet_bootstrap.cc:492] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.276619 1206 ts_tablet_manager.cc:1403] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20251214 20:52:32.277382 1206 raft_consensus.cc:359] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.277551 1206 raft_consensus.cc:385] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.277599 1206 raft_consensus.cc:740] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.282418 1203 tablet_bootstrap.cc:492] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.282591 1203 ts_tablet_manager.cc:1403] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.004s
I20251214 20:52:32.283386 2847 raft_consensus.cc:493] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.283485 2735 raft_consensus.cc:493] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.283500 1206 consensus_queue.cc:260] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.283569 2735 raft_consensus.cc:515] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.283520 2847 raft_consensus.cc:515] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.284206 2735 leader_election.cc:290] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.284214 2847 leader_election.cc:290] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.284386 1203 raft_consensus.cc:359] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.284515 1203 raft_consensus.cc:385] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.284561 1203 raft_consensus.cc:740] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.284735 1203 consensus_queue.cc:260] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.289206 1203 ts_tablet_manager.cc:1434] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:32.289196 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "088c72a6f5014b6fbc6ee060b6b7a590" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.289566 1203 tablet_bootstrap.cc:492] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.289583 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d01db63fd5d2474a9defdce0b98fc29d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.289973 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d01db63fd5d2474a9defdce0b98fc29d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.291352 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "088c72a6f5014b6fbc6ee060b6b7a590" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.291473 1203 tablet_bootstrap.cc:654] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:32.291445 989 leader_election.cc:343] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:32.291694 989 leader_election.cc:343] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.292654 1206 ts_tablet_manager.cc:1434] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
W20251214 20:52:32.292794 991 leader_election.cc:343] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.292927 991 leader_election.cc:304] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:32.293334 991 leader_election.cc:343] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.293416 991 leader_election.cc:304] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.294169 2735 raft_consensus.cc:2749] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.294330 2735 raft_consensus.cc:2749] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.294343 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 573) took 1562 ms. Trace:
I20251214 20:52:32.294579 1856 rpcz_store.cc:276] 1214 20:52:30.731830 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.741063 (+ 9233us) service_pool.cc:225] Handling call
1214 20:52:32.194396 (+1453333us) ts_tablet_manager.cc:1151] Deleting tablet e90d11d69b8d4fd49150aa67a8190e5b
1214 20:52:32.194403 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.222675 (+ 28272us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.241932 (+ 19257us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.294324 (+ 52392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1453110}
I20251214 20:52:32.295323 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet e90d11d69b8d4fd49150aa67a8190e5b (table default.Table_c5bdb81886bf4361af53fd1a3b3b0b16 [id=b3df2580f4a74189aa06f021d6b4ce71]) successfully deleted
I20251214 20:52:32.295846 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 591) took 1428 ms. Trace:
I20251214 20:52:32.298496 1856 rpcz_store.cc:276] 1214 20:52:30.866979 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.870862 (+ 3883us) service_pool.cc:225] Handling call
1214 20:52:32.295713 (+1424851us) ts_tablet_manager.cc:1151] Deleting tablet 2f75657d7bc44f6393c55ab119eb7dc2
1214 20:52:32.295725 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.295829 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1424605}
I20251214 20:52:32.299109 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 592) took 1417 ms. Trace:
W20251214 20:52:32.300361 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 2f75657d7bc44f6393c55ab119eb7dc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f75657d7bc44f6393c55ab119eb7dc2 already in progress: creating tablet
I20251214 20:52:32.301033 1206 tablet_bootstrap.cc:492] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.303179 1856 rpcz_store.cc:276] 1214 20:52:30.881831 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.881899 (+ 68us) service_pool.cc:225] Handling call
1214 20:52:32.298993 (+1417094us) ts_tablet_manager.cc:1151] Deleting tablet 348d8dcef97042279bc0d5ff31d3c137
1214 20:52:32.299004 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.299093 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1416870}
W20251214 20:52:32.303784 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 348d8dcef97042279bc0d5ff31d3c137 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 348d8dcef97042279bc0d5ff31d3c137 already in progress: creating tablet
I20251214 20:52:32.304318 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 593) took 1417 ms. Trace:
I20251214 20:52:32.304440 1856 rpcz_store.cc:276] 1214 20:52:30.886721 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.886794 (+ 73us) service_pool.cc:225] Handling call
1214 20:52:32.304195 (+1417401us) ts_tablet_manager.cc:1151] Deleting tablet 1def5f0de5ce4eb0bed36f626c4f0683
1214 20:52:32.304206 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.304301 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1417142}
W20251214 20:52:32.304947 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 1def5f0de5ce4eb0bed36f626c4f0683 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1def5f0de5ce4eb0bed36f626c4f0683 already in progress: creating tablet
W20251214 20:52:32.305686 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 0b7dd0ab78534a34ba342ec6e4fae129 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b7dd0ab78534a34ba342ec6e4fae129 already in progress: creating tablet
W20251214 20:52:32.305984 442 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 5b68367f902f4be386788a8d138b5762 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b68367f902f4be386788a8d138b5762 already in progress: creating tablet
I20251214 20:52:32.308049 1206 tablet_bootstrap.cc:654] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.311933 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 420) took 3881 ms. Trace:
I20251214 20:52:32.312059 1982 rpcz_store.cc:276] 1214 20:52:28.430818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.431175 (+ 357us) service_pool.cc:225] Handling call
1214 20:52:32.218738 (+3787563us) ts_tablet_manager.cc:1151] Deleting tablet bf8643c7e4094f529cd55656c7b3edb6
1214 20:52:32.218747 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.269177 (+ 50430us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.272675 (+ 3498us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.311904 (+ 39229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3775264}
I20251214 20:52:32.312429 1982 tablet_replica.cc:333] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.312628 1982 raft_consensus.cc:2243] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.312680 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet bf8643c7e4094f529cd55656c7b3edb6 (table default.Table_1bba0cb42a0c4aa986d55fdc492ffd46 [id=ce55ad7e80fe40e99895c52abb55e986]) successfully deleted
I20251214 20:52:32.312798 1982 raft_consensus.cc:2272] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.313390 1982 ts_tablet_manager.cc:1916] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.315584 1856 tablet_replica.cc:333] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.315830 1856 raft_consensus.cc:2243] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.315996 1856 raft_consensus.cc:2272] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.316599 1856 ts_tablet_manager.cc:1916] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.319485 1202 tablet_bootstrap.cc:492] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.319657 1202 ts_tablet_manager.cc:1403] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20251214 20:52:32.320459 1202 raft_consensus.cc:359] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.320653 1202 raft_consensus.cc:385] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.320708 1202 raft_consensus.cc:740] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.320953 1202 consensus_queue.cc:260] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.323835 1202 ts_tablet_manager.cc:1434] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:32.324170 1202 tablet_bootstrap.cc:492] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.329675 1202 tablet_bootstrap.cc:654] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.332680 1977 ts_tablet_manager.cc:1929] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251214 20:52:32.332862 1977 log.cc:1199] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/57f2314d33fa474ca36dafe0a0eae5ae
I20251214 20:52:32.333320 1977 ts_tablet_manager.cc:1950] T 57f2314d33fa474ca36dafe0a0eae5ae P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.334239 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 434) took 3525 ms. Trace:
I20251214 20:52:32.334336 1977 rpcz_store.cc:276] 1214 20:52:28.809135 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.809550 (+ 415us) service_pool.cc:225] Handling call
1214 20:52:32.207658 (+3398108us) ts_tablet_manager.cc:1151] Deleting tablet 57f2314d33fa474ca36dafe0a0eae5ae
1214 20:52:32.207668 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.238913 (+ 31245us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.306180 (+ 67267us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.334218 (+ 28038us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3397962}
I20251214 20:52:32.334698 1977 tablet_replica.cc:333] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.334898 1977 raft_consensus.cc:2243] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 2 LEADER]: Raft consensus shutting down.
I20251214 20:52:32.335172 1977 pending_rounds.cc:70] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:32.335230 1977 pending_rounds.cc:77] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7232493781450534912 op_type: NO_OP noop_request { }
I20251214 20:52:32.335208 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 57f2314d33fa474ca36dafe0a0eae5ae (table default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c]) successfully deleted
I20251214 20:52:32.335338 1977 raft_consensus.cc:2889] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:32.335422 1977 raft_consensus.cc:2272] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.340168 1977 ts_tablet_manager.cc:1916] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.341507 2847 raft_consensus.cc:493] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.341691 2847 raft_consensus.cc:515] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.342435 2847 leader_election.cc:290] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.343416 1206 tablet_bootstrap.cc:492] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.343338 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "140667afe42f42659a8cd38ed2517201" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.343811 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "140667afe42f42659a8cd38ed2517201" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:32.344238 989 leader_election.cc:343] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:32.344374 991 leader_election.cc:343] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.344489 991 leader_election.cc:304] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.344764 1206 ts_tablet_manager.cc:1403] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.001s
I20251214 20:52:32.344870 2735 raft_consensus.cc:2749] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.345479 1206 raft_consensus.cc:359] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.345723 1206 raft_consensus.cc:385] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.345813 1206 raft_consensus.cc:740] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.346086 1206 consensus_queue.cc:260] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.351562 1206 ts_tablet_manager.cc:1434] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251214 20:52:32.351588 2787 raft_consensus.cc:493] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.351711 2787 raft_consensus.cc:515] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.352002 1206 tablet_bootstrap.cc:492] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.352325 2787 leader_election.cc:290] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.353171 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d157ac8f74eb7911df05ab5e4a3d5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.353268 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d157ac8f74eb7911df05ab5e4a3d5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:32.353497 1050 raft_consensus.cc:2468] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:32.354338 1202 tablet_bootstrap.cc:492] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.354535 1202 ts_tablet_manager.cc:1403] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
W20251214 20:52:32.354955 848 leader_election.cc:343] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.355479 1202 raft_consensus.cc:359] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.355677 1202 raft_consensus.cc:385] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.355773 1202 raft_consensus.cc:740] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.355852 851 leader_election.cc:304] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.356074 1202 consensus_queue.cc:260] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.356420 2777 raft_consensus.cc:2804] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:32.356519 2777 raft_consensus.cc:493] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:32.356581 2777 raft_consensus.cc:3060] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.358444 1202 ts_tablet_manager.cc:1434] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:32.358749 1202 tablet_bootstrap.cc:492] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.361203 1202 tablet_bootstrap.cc:654] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.361555 2777 raft_consensus.cc:515] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.362236 2777 leader_election.cc:290] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.364560 1982 ts_tablet_manager.cc:1929] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251214 20:52:32.364701 1982 log.cc:1199] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/b70579ddf96f4499b4e4f235e8f69d67
I20251214 20:52:32.365092 1982 ts_tablet_manager.cc:1950] T b70579ddf96f4499b4e4f235e8f69d67 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.365271 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d157ac8f74eb7911df05ab5e4a3d5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:32.365940 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 418) took 3949 ms. Trace:
I20251214 20:52:32.366036 1982 rpcz_store.cc:276] 1214 20:52:28.416121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.416615 (+ 494us) service_pool.cc:225] Handling call
1214 20:52:32.312373 (+3895758us) ts_tablet_manager.cc:1151] Deleting tablet b70579ddf96f4499b4e4f235e8f69d67
1214 20:52:32.312386 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.345736 (+ 33350us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.364410 (+ 18674us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.365918 (+ 1508us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24,"tablet-delete.queue_time_us":3867999}
I20251214 20:52:32.366546 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet b70579ddf96f4499b4e4f235e8f69d67 (table default.Table_3a068835084e406eb4047a827faeb5d7 [id=bdb017130eed4d76b635603560c91d41]) successfully deleted
I20251214 20:52:32.366657 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d157ac8f74eb7911df05ab5e4a3d5" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:32.366868 1050 raft_consensus.cc:3060] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.367105 1977 ts_tablet_manager.cc:1929] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20251214 20:52:32.367244 1977 log.cc:1199] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/98b7cac3930a4a9eb16ac413c8f1e092
I20251214 20:52:32.367551 1977 ts_tablet_manager.cc:1950] T 98b7cac3930a4a9eb16ac413c8f1e092 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.367806 1206 tablet_bootstrap.cc:654] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.368395 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 435) took 3559 ms. Trace:
I20251214 20:52:32.368479 1977 rpcz_store.cc:276] 1214 20:52:28.809230 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.809724 (+ 494us) service_pool.cc:225] Handling call
1214 20:52:32.334645 (+3524921us) ts_tablet_manager.cc:1151] Deleting tablet 98b7cac3930a4a9eb16ac413c8f1e092
1214 20:52:32.334655 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.353115 (+ 18460us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.366960 (+ 13845us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.368371 (+ 1411us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3524801}
I20251214 20:52:32.369058 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 98b7cac3930a4a9eb16ac413c8f1e092 (table default.Table_e5f3db562155448e91e2e4314254cbe6 [id=826a2666d167435c8fdfd9ab8b8819e6]) successfully deleted
W20251214 20:52:32.369544 848 leader_election.cc:343] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.369705 1050 raft_consensus.cc:2468] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:32.371186 1203 tablet_bootstrap.cc:492] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.371332 1203 ts_tablet_manager.cc:1403] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.000s
I20251214 20:52:32.371874 851 leader_election.cc:304] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.372182 2777 raft_consensus.cc:2804] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:32.372319 2777 raft_consensus.cc:697] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:32.372573 2777 consensus_queue.cc:237] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:32.375308 440 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 9bd89709abbc49809a2cdb6d0c29fdb8 (table default.Table_c3aa8e3c31cb41dc8281962e7b5c46e1 [id=687eb544b4114bad8d2deb76388e2022], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9bd89709abbc49809a2cdb6d0c29fdb8 consensus state has no leader
I20251214 20:52:32.377216 1203 raft_consensus.cc:359] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.377409 1203 raft_consensus.cc:385] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.377456 1203 raft_consensus.cc:740] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.378082 1203 consensus_queue.cc:260] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.378635 1203 ts_tablet_manager.cc:1434] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251214 20:52:32.379698 1203 tablet_bootstrap.cc:492] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.390671 1202 tablet_bootstrap.cc:492] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.390838 1202 ts_tablet_manager.cc:1403] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20251214 20:52:32.391575 1202 raft_consensus.cc:359] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.391749 1202 raft_consensus.cc:385] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.391799 1202 raft_consensus.cc:740] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.392004 1202 consensus_queue.cc:260] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.392608 1202 ts_tablet_manager.cc:1434] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:32.392884 1202 tablet_bootstrap.cc:492] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.395895 1202 tablet_bootstrap.cc:654] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.397027 773 raft_consensus.cc:3060] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.400010 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 442) took 3524 ms. Trace:
I20251214 20:52:32.400955 2735 raft_consensus.cc:493] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.401147 2735 raft_consensus.cc:515] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.401873 2735 leader_election.cc:290] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.402670 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe24e4f4e74040249068505058c15622" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.403302 989 leader_election.cc:343] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.404018 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe24e4f4e74040249068505058c15622" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:32.404815 991 leader_election.cc:343] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.404953 991 leader_election.cc:304] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.405377 2735 raft_consensus.cc:2749] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.400131 1977 rpcz_store.cc:276] 1214 20:52:28.875405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.875461 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:32.399823 (+3524362us) ts_tablet_manager.cc:1151] Deleting tablet 1e185d107619414a8bcc15ad496547dc
1214 20:52:32.399838 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.399981 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3524167}
W20251214 20:52:32.406519 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 1e185d107619414a8bcc15ad496547dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e185d107619414a8bcc15ad496547dc already in progress: creating tablet
I20251214 20:52:32.407226 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 446) took 3465 ms. Trace:
I20251214 20:52:32.407336 1977 rpcz_store.cc:276] 1214 20:52:28.941906 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.942811 (+ 905us) service_pool.cc:225] Handling call
1214 20:52:32.407067 (+3464256us) ts_tablet_manager.cc:1151] Deleting tablet 3915abc53b084854912a8e4119eefc7e
1214 20:52:32.407081 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.407200 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3463997}
W20251214 20:52:32.407793 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 3915abc53b084854912a8e4119eefc7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3915abc53b084854912a8e4119eefc7e already in progress: creating tablet
I20251214 20:52:32.408156 1203 tablet_bootstrap.cc:654] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.410501 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 447) took 3426 ms. Trace:
I20251214 20:52:32.410612 1977 rpcz_store.cc:276] 1214 20:52:28.984536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.984796 (+ 260us) service_pool.cc:225] Handling call
1214 20:52:32.410356 (+3425560us) ts_tablet_manager.cc:1151] Deleting tablet 6b389a63a58d438db41a343db6e04a88
1214 20:52:32.410367 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.410475 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3425242}
W20251214 20:52:32.411108 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 6b389a63a58d438db41a343db6e04a88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b389a63a58d438db41a343db6e04a88 already in progress: creating tablet
I20251214 20:52:32.411955 1206 tablet_bootstrap.cc:492] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.412312 1856 ts_tablet_manager.cc:1929] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.412434 1856 log.cc:1199] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/274185a4115b458e8e9a376bdbcc870b
I20251214 20:52:32.412708 1206 ts_tablet_manager.cc:1403] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.006s
I20251214 20:52:32.412757 1856 ts_tablet_manager.cc:1950] T 274185a4115b458e8e9a376bdbcc870b P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.413622 1206 raft_consensus.cc:359] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.413858 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 274185a4115b458e8e9a376bdbcc870b (table default.Table_e82b745b92174d32a0d7bc8f779a6558 [id=43a932a17bd545f282a790e6c094a4e6]) successfully deleted
I20251214 20:52:32.414796 1206 raft_consensus.cc:385] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.415048 1206 raft_consensus.cc:740] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.415668 1206 consensus_queue.cc:260] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.416723 1206 ts_tablet_manager.cc:1434] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251214 20:52:32.417454 1206 tablet_bootstrap.cc:492] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.419574 1206 tablet_bootstrap.cc:654] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:32.428355 848 consensus_peers.cc:597] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:32.422123 1982 tablet_replica.cc:333] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.434423 1977 tablet_replica.cc:333] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.434738 2735 raft_consensus.cc:493] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.434932 2735 raft_consensus.cc:515] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.435670 2735 leader_election.cc:290] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.436316 1206 tablet_bootstrap.cc:492] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.436456 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d7c7b3f62764ff497f4d70b53eed7de" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.437283 989 leader_election.cc:343] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.437350 1206 ts_tablet_manager.cc:1403] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20251214 20:52:32.438081 1982 raft_consensus.cc:2243] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.438149 2688 raft_consensus.cc:493] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.438314 1982 raft_consensus.cc:2272] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.438369 2688 raft_consensus.cc:515] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:32.434669 1977 raft_consensus.cc:2243] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.436957 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d7c7b3f62764ff497f4d70b53eed7de" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.438699 1977 raft_consensus.cc:2272] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.438927 1982 ts_tablet_manager.cc:1916] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:32.439318 991 leader_election.cc:343] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.439421 991 leader_election.cc:304] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.439461 2688 leader_election.cc:290] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:32.440008 2735 raft_consensus.cc:2749] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.440712 1334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 6 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" is_pre_election: true
I20251214 20:52:32.441030 1334 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c27224391721467a8f46e467a980b37a in term 5.
I20251214 20:52:32.441321 2397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 6 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
I20251214 20:52:32.441625 2397 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c27224391721467a8f46e467a980b37a in term 5.
I20251214 20:52:32.442194 441 leader_election.cc:304] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, c27224391721467a8f46e467a980b37a; no voters:
I20251214 20:52:32.445958 2688 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251214 20:52:32.446130 2688 raft_consensus.cc:493] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:32.446213 2688 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 5 FOLLOWER]: Advancing to term 6
I20251214 20:52:32.447753 2777 raft_consensus.cc:493] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.447935 2777 raft_consensus.cc:515] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.449002 2777 leader_election.cc:290] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.449183 1203 tablet_bootstrap.cc:492] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.449112 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ebfc373fbf41b1920c271a15302484" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:32.449330 1051 raft_consensus.cc:2468] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:32.449353 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ebfc373fbf41b1920c271a15302484" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.449745 851 leader_election.cc:304] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:32.450474 1977 ts_tablet_manager.cc:1916] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.454180 2863 raft_consensus.cc:2804] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:32.454269 2863 raft_consensus.cc:493] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:32.454346 2863 raft_consensus.cc:3060] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.457617 2863 raft_consensus.cc:515] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.458264 2863 leader_election.cc:290] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.449350 1203 ts_tablet_manager.cc:1403] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.001s
I20251214 20:52:32.458803 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ebfc373fbf41b1920c271a15302484" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:32.459326 1203 raft_consensus.cc:359] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.459501 1203 raft_consensus.cc:385] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.459544 1203 raft_consensus.cc:740] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:32.459491 848 leader_election.cc:343] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.459784 1203 consensus_queue.cc:260] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.461113 2735 raft_consensus.cc:493] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.461273 2735 raft_consensus.cc:515] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.461628 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ebfc373fbf41b1920c271a15302484" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:32.461875 1051 raft_consensus.cc:3060] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.462415 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfe50a6219134431b0a025317932609b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.462913 989 leader_election.cc:343] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.463740 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfe50a6219134431b0a025317932609b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.463855 1203 ts_tablet_manager.cc:1434] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:32.464676 1051 raft_consensus.cc:2468] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:32.461880 2735 leader_election.cc:290] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:32.466703 991 leader_election.cc:343] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.466820 991 leader_election.cc:304] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.468240 2735 raft_consensus.cc:2749] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.468719 1203 tablet_bootstrap.cc:492] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.470217 1856 tablet_replica.cc:333] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.470446 1856 raft_consensus.cc:2243] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.470609 1856 raft_consensus.cc:2272] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.471206 1856 ts_tablet_manager.cc:1916] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.471588 851 leader_election.cc:304] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.473776 1202 tablet_bootstrap.cc:492] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.473907 1202 ts_tablet_manager.cc:1403] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.081s user 0.013s sys 0.015s
I20251214 20:52:32.474575 1202 raft_consensus.cc:359] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.474746 1202 raft_consensus.cc:385] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.474795 1202 raft_consensus.cc:740] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.474983 1202 consensus_queue.cc:260] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.475309 2777 raft_consensus.cc:2804] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:32.475469 2777 raft_consensus.cc:697] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:32.475493 1202 ts_tablet_manager.cc:1434] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:32.475761 1202 tablet_bootstrap.cc:492] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.475725 2777 consensus_queue.cc:237] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.477173 1202 tablet_bootstrap.cc:654] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.478502 2688 raft_consensus.cc:515] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:32.480871 1982 ts_tablet_manager.cc:1929] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.481011 1982 log.cc:1199] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/fe0c3c8a1d5b46cfacd35d3f0d82e432
I20251214 20:52:32.481333 1982 ts_tablet_manager.cc:1950] T fe0c3c8a1d5b46cfacd35d3f0d82e432 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.482164 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 422) took 3747 ms. Trace:
I20251214 20:52:32.482270 1982 rpcz_store.cc:276] 1214 20:52:28.734386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.734460 (+ 74us) service_pool.cc:225] Handling call
1214 20:52:32.422049 (+3687589us) ts_tablet_manager.cc:1151] Deleting tablet fe0c3c8a1d5b46cfacd35d3f0d82e432
1214 20:52:32.422064 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.471095 (+ 49031us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.480704 (+ 9609us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.482140 (+ 1436us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3687378}
I20251214 20:52:32.482620 1982 tablet_replica.cc:333] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.482825 1982 raft_consensus.cc:2243] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.482957 1982 raft_consensus.cc:2272] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.483402 1203 tablet_bootstrap.cc:654] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.483510 1982 ts_tablet_manager.cc:1916] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.483572 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet fe0c3c8a1d5b46cfacd35d3f0d82e432 (table default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]) successfully deleted
I20251214 20:52:32.484388 1202 tablet_bootstrap.cc:492] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.484513 1202 ts_tablet_manager.cc:1403] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:32.485158 1202 raft_consensus.cc:359] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.485292 1202 raft_consensus.cc:385] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.485333 1202 raft_consensus.cc:740] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.485415 1206 raft_consensus.cc:359] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.485491 1202 consensus_queue.cc:260] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.485569 1206 raft_consensus.cc:385] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.485620 1206 raft_consensus.cc:740] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.485849 1206 consensus_queue.cc:260] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.485956 1202 ts_tablet_manager.cc:1434] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 20:52:32.486218 1202 tablet_bootstrap.cc:492] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.488116 1202 tablet_bootstrap.cc:654] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.488106 776 raft_consensus.cc:3060] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.486455 1206 ts_tablet_manager.cc:1434] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20251214 20:52:32.491797 1206 tablet_bootstrap.cc:492] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.493492 1206 tablet_bootstrap.cc:654] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.499660 1202 tablet_bootstrap.cc:492] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.499825 1202 ts_tablet_manager.cc:1403] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20251214 20:52:32.500510 1202 raft_consensus.cc:359] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.500669 1202 raft_consensus.cc:385] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.500743 1202 raft_consensus.cc:740] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.500962 1202 consensus_queue.cc:260] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.501658 1202 ts_tablet_manager.cc:1434] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:32.503160 2787 consensus_queue.cc:1048] T e6bdfbe4d15345de8be5de00bfe4ca07 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:32.503696 1202 tablet_bootstrap.cc:492] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.506181 1202 tablet_bootstrap.cc:654] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.508951 1203 tablet_bootstrap.cc:492] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.509099 1203 ts_tablet_manager.cc:1403] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20251214 20:52:32.509855 1203 raft_consensus.cc:359] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.510036 1203 raft_consensus.cc:385] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.510090 1203 raft_consensus.cc:740] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.514827 1977 ts_tablet_manager.cc:1929] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.514982 1977 log.cc:1199] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/25dcf491653147e7831217c1ec7a2e0c
I20251214 20:52:32.515347 1977 ts_tablet_manager.cc:1950] T 25dcf491653147e7831217c1ec7a2e0c P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.516474 1334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 6 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4"
I20251214 20:52:32.516731 1334 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 5 FOLLOWER]: Advancing to term 6
I20251214 20:52:32.519784 1334 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c27224391721467a8f46e467a980b37a in term 6.
I20251214 20:52:32.520331 441 leader_election.cc:304] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, c27224391721467a8f46e467a980b37a; no voters:
I20251214 20:52:32.520661 2779 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 6 FOLLOWER]: Leader election won for term 6
I20251214 20:52:32.520759 1202 tablet_bootstrap.cc:492] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.520910 1202 ts_tablet_manager.cc:1403] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.006s
I20251214 20:52:32.521646 1202 raft_consensus.cc:359] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.521816 1202 raft_consensus.cc:385] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.521853 1202 raft_consensus.cc:740] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.522035 1202 consensus_queue.cc:260] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.522359 2397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c27224391721467a8f46e467a980b37a" candidate_term: 6 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad"
I20251214 20:52:32.522573 2397 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 5 FOLLOWER]: Advancing to term 6
I20251214 20:52:32.524890 1202 ts_tablet_manager.cc:1434] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:32.525187 1202 tablet_bootstrap.cc:492] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.525353 2397 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c27224391721467a8f46e467a980b37a in term 6.
I20251214 20:52:32.525751 1206 tablet_bootstrap.cc:492] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.525926 1206 ts_tablet_manager.cc:1403] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.001s
I20251214 20:52:32.526785 1202 tablet_bootstrap.cc:654] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.526767 1206 raft_consensus.cc:359] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.526957 1206 raft_consensus.cc:385] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.527021 1206 raft_consensus.cc:740] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.527413 1203 consensus_queue.cc:260] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.527817 2779 raft_consensus.cc:697] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [term 6 LEADER]: Becoming Leader. State: Replica: c27224391721467a8f46e467a980b37a, State: Running, Role: LEADER
I20251214 20:52:32.528025 1203 ts_tablet_manager.cc:1434] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20251214 20:52:32.528213 2779 consensus_queue.cc:237] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 955, Committed index: 955, Last appended: 4.955, Last appended by leader: 955, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:32.529920 2688 leader_election.cc:290] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [CANDIDATE]: Term 6 election: Requested vote from peers e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:32.530318 1856 ts_tablet_manager.cc:1929] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.530434 1856 log.cc:1199] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/fc7353593bda4fceaa389c393ddc19c6
I20251214 20:52:32.530728 1856 ts_tablet_manager.cc:1950] T fc7353593bda4fceaa389c393ddc19c6 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.531253 1203 tablet_bootstrap.cc:492] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.533102 1203 tablet_bootstrap.cc:654] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.534406 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 449) took 3522 ms. Trace:
I20251214 20:52:32.534518 1977 rpcz_store.cc:276] 1214 20:52:29.011889 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.016230 (+ 4341us) service_pool.cc:225] Handling call
1214 20:52:32.434342 (+3418112us) ts_tablet_manager.cc:1151] Deleting tablet 25dcf491653147e7831217c1ec7a2e0c
1214 20:52:32.434357 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.493900 (+ 59543us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.514684 (+ 20784us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.534373 (+ 19689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3416523}
I20251214 20:52:32.535118 441 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 25dcf491653147e7831217c1ec7a2e0c (table default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7]) successfully deleted
I20251214 20:52:32.538417 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 450) took 3511 ms. Trace:
I20251214 20:52:32.539696 1206 consensus_queue.cc:260] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.540244 1977 rpcz_store.cc:276] 1214 20:52:29.027385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.027820 (+ 435us) service_pool.cc:225] Handling call
1214 20:52:32.538264 (+3510444us) ts_tablet_manager.cc:1151] Deleting tablet a88207c8cc5f4a6886de36c82c6b4c2f
1214 20:52:32.538275 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.538395 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3510227}
W20251214 20:52:32.540789 441 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet a88207c8cc5f4a6886de36c82c6b4c2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a88207c8cc5f4a6886de36c82c6b4c2f already in progress: creating tablet
I20251214 20:52:32.541574 1206 ts_tablet_manager.cc:1434] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20251214 20:52:32.542027 1977 tablet_replica.cc:333] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.542462 1977 raft_consensus.cc:2243] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:32.500594 1982 ts_tablet_manager.cc:1929] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.542778 1982 log.cc:1199] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/98105bceda9b44bf893f82a4234a3c81
I20251214 20:52:32.543078 1206 tablet_bootstrap.cc:492] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.543478 1977 pending_rounds.cc:70] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:32.543529 1977 pending_rounds.cc:77] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493769582223360 op_type: NO_OP noop_request { }
I20251214 20:52:32.543632 1977 raft_consensus.cc:2889] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:32.543715 1977 raft_consensus.cc:2272] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.543172 1982 ts_tablet_manager.cc:1950] T 98105bceda9b44bf893f82a4234a3c81 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.545140 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 423) took 3790 ms. Trace:
I20251214 20:52:32.547116 442 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet fc7353593bda4fceaa389c393ddc19c6 (table default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589]) successfully deleted
I20251214 20:52:32.549559 1977 ts_tablet_manager.cc:1916] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.552449 1856 tablet_replica.cc:333] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.560719 1856 raft_consensus.cc:2243] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.561645 1856 raft_consensus.cc:2272] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.563504 1856 ts_tablet_manager.cc:1916] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.562839 1203 tablet_bootstrap.cc:492] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.567106 1203 ts_tablet_manager.cc:1403] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20251214 20:52:32.568105 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27224391721467a8f46e467a980b37a. Latest consensus state: current_term: 6 leader_uuid: "c27224391721467a8f46e467a980b37a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:32.570127 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P c27224391721467a8f46e467a980b37a [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:32.568204 1203 raft_consensus.cc:359] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.574955 1203 raft_consensus.cc:385] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.575060 1203 raft_consensus.cc:740] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.576483 2883 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 20:52:32.576813 1982 rpcz_store.cc:276] 1214 20:52:28.754793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.754879 (+ 86us) service_pool.cc:225] Handling call
1214 20:52:32.482565 (+3727686us) ts_tablet_manager.cc:1151] Deleting tablet 98105bceda9b44bf893f82a4234a3c81
1214 20:52:32.482577 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.497500 (+ 14923us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.500488 (+ 2988us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.545118 (+ 44630us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3701106}
I20251214 20:52:32.577346 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 431) took 3703 ms. Trace:
I20251214 20:52:32.577426 1982 rpcz_store.cc:276] 1214 20:52:28.874255 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.875971 (+ 1716us) service_pool.cc:225] Handling call
1214 20:52:32.577173 (+3701202us) ts_tablet_manager.cc:1151] Deleting tablet 1e185d107619414a8bcc15ad496547dc
1214 20:52:32.577186 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.577330 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3700894}
I20251214 20:52:32.577693 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 434) took 3633 ms. Trace:
I20251214 20:52:32.577755 1982 rpcz_store.cc:276] 1214 20:52:28.944007 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.947459 (+ 3452us) service_pool.cc:225] Handling call
1214 20:52:32.577607 (+3630148us) ts_tablet_manager.cc:1151] Deleting tablet 3915abc53b084854912a8e4119eefc7e
1214 20:52:32.577615 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.577680 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3629909}
I20251214 20:52:32.578030 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 436) took 3593 ms. Trace:
I20251214 20:52:32.578100 1982 rpcz_store.cc:276] 1214 20:52:28.984446 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:28.984535 (+ 89us) service_pool.cc:225] Handling call
1214 20:52:32.577944 (+3593409us) ts_tablet_manager.cc:1151] Deleting tablet 6b389a63a58d438db41a343db6e04a88
1214 20:52:32.577951 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.578016 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3593192}
I20251214 20:52:32.546034 1206 tablet_bootstrap.cc:654] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.555447 2819 raft_consensus.cc:493] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.578742 2819 raft_consensus.cc:515] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.579483 2819 leader_election.cc:290] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.580353 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1dca369dc5f54a4ea1783e334c5670d6" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.580812 2874 tablet.cc:1722] T 9da5e98c6f4843449886176e9fcfa8e1 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251214 20:52:32.584060 713 leader_election.cc:343] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 1dca369dc5f54a4ea1783e334c5670d6
I20251214 20:52:32.584192 713 leader_election.cc:304] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.578421 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 437) took 3572 ms. Trace:
I20251214 20:52:32.584331 1982 rpcz_store.cc:276] 1214 20:52:29.006202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.006319 (+ 117us) service_pool.cc:225] Handling call
1214 20:52:32.578311 (+3571992us) ts_tablet_manager.cc:1151] Deleting tablet 25dcf491653147e7831217c1ec7a2e0c
1214 20:52:32.578318 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.578387 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3571723}
I20251214 20:52:32.584666 2819 raft_consensus.cc:2749] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.577389 439 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 98105bceda9b44bf893f82a4234a3c81 (table default.Table_cd43be14c9f84889bd6e03167dd35f6c [id=ece0780edb0f41b787b46b290c4c0515]) successfully deleted
W20251214 20:52:32.585709 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 1e185d107619414a8bcc15ad496547dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e185d107619414a8bcc15ad496547dc already in progress: creating tablet
I20251214 20:52:32.585726 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 439) took 3549 ms. Trace:
I20251214 20:52:32.585803 1982 rpcz_store.cc:276] 1214 20:52:29.035854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.036156 (+ 302us) service_pool.cc:225] Handling call
1214 20:52:32.585605 (+3549449us) ts_tablet_manager.cc:1151] Deleting tablet a88207c8cc5f4a6886de36c82c6b4c2f
1214 20:52:32.585615 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.585709 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3549251}
W20251214 20:52:32.585915 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 3915abc53b084854912a8e4119eefc7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3915abc53b084854912a8e4119eefc7e already in progress: creating tablet
W20251214 20:52:32.586090 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 6b389a63a58d438db41a343db6e04a88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b389a63a58d438db41a343db6e04a88 already in progress: creating tablet
W20251214 20:52:32.586236 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 25dcf491653147e7831217c1ec7a2e0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25dcf491653147e7831217c1ec7a2e0c already in progress: creating tablet
I20251214 20:52:32.586241 1203 consensus_queue.cc:260] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:32.586401 439 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet a88207c8cc5f4a6886de36c82c6b4c2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a88207c8cc5f4a6886de36c82c6b4c2f already in progress: creating tablet
I20251214 20:52:32.586813 1203 ts_tablet_manager.cc:1434] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251214 20:52:32.587092 1203 tablet_bootstrap.cc:492] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.589006 1203 tablet_bootstrap.cc:654] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.598546 2735 raft_consensus.cc:493] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.598721 2735 raft_consensus.cc:515] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.599391 2735 leader_election.cc:290] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.600121 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae948ffc81764692b92045f5befe0184" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:32.600785 991 leader_election.cc:343] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.605271 1977 ts_tablet_manager.cc:1929] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.605409 1977 log.cc:1199] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/8d9f74fb2c6244a6887e3c922262bbab
I20251214 20:52:32.605749 1977 ts_tablet_manager.cc:1950] T 8d9f74fb2c6244a6887e3c922262bbab P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.609874 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae948ffc81764692b92045f5befe0184" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.610570 989 leader_election.cc:343] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.610713 989 leader_election.cc:304] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.611145 2735 raft_consensus.cc:2749] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.611690 2787 raft_consensus.cc:493] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.611824 2787 raft_consensus.cc:515] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.612443 2787 leader_election.cc:290] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.612540 1202 tablet_bootstrap.cc:492] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.612704 1202 ts_tablet_manager.cc:1403] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.088s user 0.011s sys 0.018s
I20251214 20:52:32.613322 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65e79db74fa046698507c83416ee1669" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:32.613348 1202 raft_consensus.cc:359] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.613517 1202 raft_consensus.cc:385] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.613515 1050 raft_consensus.cc:2468] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:32.613512 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65e79db74fa046698507c83416ee1669" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.613575 1202 raft_consensus.cc:740] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.613775 1202 consensus_queue.cc:260] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:32.613932 848 leader_election.cc:343] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.614225 1202 ts_tablet_manager.cc:1434] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 20:52:32.614269 851 leader_election.cc:304] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.614516 1202 tablet_bootstrap.cc:492] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.614835 2787 raft_consensus.cc:2804] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:32.614944 2787 raft_consensus.cc:493] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:32.615015 2787 raft_consensus.cc:3060] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.616283 1202 tablet_bootstrap.cc:654] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.619297 2787 raft_consensus.cc:515] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.621177 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65e79db74fa046698507c83416ee1669" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:32.621382 1050 raft_consensus.cc:3060] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.622540 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65e79db74fa046698507c83416ee1669" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:32.624095 848 leader_election.cc:343] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.624221 1202 tablet_bootstrap.cc:492] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.624235 1050 raft_consensus.cc:2468] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:32.624327 1202 ts_tablet_manager.cc:1403] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
W20251214 20:52:32.624452 848 leader_election.cc:343] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.624686 851 leader_election.cc:304] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.624943 1202 raft_consensus.cc:359] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.624991 2863 raft_consensus.cc:2804] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:32.625592 2863 raft_consensus.cc:697] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:32.626011 2863 consensus_queue.cc:237] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.625102 1202 raft_consensus.cc:385] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.626403 1202 raft_consensus.cc:740] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.626731 1202 consensus_queue.cc:260] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.630707 2787 leader_election.cc:290] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.630704 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 440) took 3583 ms. Trace:
I20251214 20:52:32.631748 2884 raft_consensus.cc:493] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.631978 2884 raft_consensus.cc:515] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.632838 2884 leader_election.cc:290] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.630812 1982 rpcz_store.cc:276] 1214 20:52:29.046851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.048782 (+ 1931us) service_pool.cc:225] Handling call
1214 20:52:32.630492 (+3581710us) ts_tablet_manager.cc:1151] Deleting tablet b2bea617954c475bac7e9b8719faa028
1214 20:52:32.630511 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.630671 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3579528}
I20251214 20:52:32.634246 1982 tablet_replica.cc:333] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.634557 1982 raft_consensus.cc:2243] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.634809 1982 raft_consensus.cc:2272] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.635489 1203 tablet_bootstrap.cc:492] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.635658 1203 ts_tablet_manager.cc:1403] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20251214 20:52:32.636251 1982 ts_tablet_manager.cc:1916] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.636499 1203 raft_consensus.cc:359] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.636682 1203 raft_consensus.cc:385] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.636741 1203 raft_consensus.cc:740] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.638064 1206 tablet_bootstrap.cc:492] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.638271 1206 ts_tablet_manager.cc:1403] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.095s user 0.002s sys 0.003s
I20251214 20:52:32.639112 1206 raft_consensus.cc:359] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.639408 1206 raft_consensus.cc:385] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.639500 1206 raft_consensus.cc:740] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.639806 1206 consensus_queue.cc:260] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.644384 1856 ts_tablet_manager.cc:1929] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.644519 1856 log.cc:1199] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/feb19ea389af4ea6bb158a0e8aaeb094
W20251214 20:52:32.647485 713 leader_election.cc:343] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 69f6108792fa4c2a9744e3248c76251f
I20251214 20:52:32.647563 1856 ts_tablet_manager.cc:1950] T feb19ea389af4ea6bb158a0e8aaeb094 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.648651 1206 ts_tablet_manager.cc:1434] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251214 20:52:32.648715 1856 tablet_replica.cc:333] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.650441 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 454) took 3539 ms. Trace:
I20251214 20:52:32.650713 1977 rpcz_store.cc:276] 1214 20:52:29.111409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.111500 (+ 91us) service_pool.cc:225] Handling call
1214 20:52:32.541975 (+3430475us) ts_tablet_manager.cc:1151] Deleting tablet 8d9f74fb2c6244a6887e3c922262bbab
1214 20:52:32.541985 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.591262 (+ 49277us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.605132 (+ 13870us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.606598 (+ 1466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3406495}
I20251214 20:52:32.651294 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 451) took 3605 ms. Trace:
I20251214 20:52:32.651616 1856 raft_consensus.cc:2243] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.652109 1206 tablet_bootstrap.cc:492] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:32.653651 712 leader_election.cc:343] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 69f6108792fa4c2a9744e3248c76251f
I20251214 20:52:32.653780 712 leader_election.cc:304] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.654306 2884 raft_consensus.cc:2749] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.656744 1206 tablet_bootstrap.cc:654] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.636965 1203 consensus_queue.cc:260] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.655551 1202 ts_tablet_manager.cc:1434] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.031s user 0.005s sys 0.006s
I20251214 20:52:32.663147 1202 tablet_bootstrap.cc:492] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.664816 1202 tablet_bootstrap.cc:654] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.668831 1977 rpcz_store.cc:276] 1214 20:52:29.045698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.045984 (+ 286us) service_pool.cc:225] Handling call
1214 20:52:32.651164 (+3605180us) ts_tablet_manager.cc:1151] Deleting tablet b2bea617954c475bac7e9b8719faa028
1214 20:52:32.651173 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.651278 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3515313}
I20251214 20:52:32.669200 1977 tablet_replica.cc:333] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.669399 1977 raft_consensus.cc:2243] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.669561 1977 raft_consensus.cc:2272] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.670130 1977 ts_tablet_manager.cc:1916] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.657780 1856 raft_consensus.cc:2272] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.662777 2735 raft_consensus.cc:493] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.671548 2735 raft_consensus.cc:515] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.671837 1856 ts_tablet_manager.cc:1916] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.671795 2889 raft_consensus.cc:493] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.672040 2889 raft_consensus.cc:515] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.672363 2735 leader_election.cc:290] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.672904 2889 leader_election.cc:290] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.673130 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b69bb4d5979f492683fc13238a668b9f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.673410 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b69bb4d5979f492683fc13238a668b9f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:32.673599 1051 raft_consensus.cc:2468] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:32.673724 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e33ba1149a44c294885d76f676160a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.673992 851 leader_election.cc:304] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:32.674175 991 leader_election.cc:343] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.674211 2889 raft_consensus.cc:2804] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:32.674279 2889 raft_consensus.cc:493] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:32.674335 2889 raft_consensus.cc:3060] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:32.674538 848 leader_election.cc:343] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.675060 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e33ba1149a44c294885d76f676160a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.675565 989 leader_election.cc:343] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.675658 989 leader_election.cc:304] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.676052 2735 raft_consensus.cc:2749] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.677333 2889 raft_consensus.cc:515] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.677785 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu pid 424
I20251214 20:52:32.677886 2889 leader_election.cc:290] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.678673 1206 tablet_bootstrap.cc:492] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.678819 1206 ts_tablet_manager.cc:1403] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20251214 20:52:32.679704 485 generic_service.cc:196] Checking for leaks (request via RPC)
I20251214 20:52:32.679837 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b69bb4d5979f492683fc13238a668b9f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:32.680068 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b69bb4d5979f492683fc13238a668b9f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:32.680279 1051 raft_consensus.cc:3060] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:32.680306 848 leader_election.cc:343] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.682032 1206 raft_consensus.cc:359] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.682193 1206 raft_consensus.cc:385] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.682245 1206 raft_consensus.cc:740] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.682668 1203 ts_tablet_manager.cc:1434] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20251214 20:52:32.682963 1203 tablet_bootstrap.cc:492] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.688134 768 raft_consensus.cc:3060] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.692359 1051 raft_consensus.cc:2468] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:32.694376 851 leader_election.cc:304] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.694831 2777 raft_consensus.cc:2804] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:32.694975 2777 raft_consensus.cc:697] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:32.695333 1856 ts_tablet_manager.cc:1929] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.695271 2777 consensus_queue.cc:237] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.695676 1856 log.cc:1199] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/e6ebfc373fbf41b1920c271a15302484
I20251214 20:52:32.696023 1856 ts_tablet_manager.cc:1950] T e6ebfc373fbf41b1920c271a15302484 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.697221 1856 tablet_replica.cc:333] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.697449 1856 raft_consensus.cc:2243] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.697621 1856 raft_consensus.cc:2272] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.698137 1856 ts_tablet_manager.cc:1916] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.703608 1856 ts_tablet_manager.cc:1929] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.703752 1856 log.cc:1199] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/b69bb4d5979f492683fc13238a668b9f
I20251214 20:52:32.704018 1856 ts_tablet_manager.cc:1950] T b69bb4d5979f492683fc13238a668b9f P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.705492 1202 tablet_bootstrap.cc:492] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.705617 1202 ts_tablet_manager.cc:1403] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.002s
I20251214 20:52:32.706278 1202 raft_consensus.cc:359] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.706429 1202 raft_consensus.cc:385] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.706497 1202 raft_consensus.cc:740] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.706666 1202 consensus_queue.cc:260] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.707190 1202 ts_tablet_manager.cc:1434] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:32.707970 1206 consensus_queue.cc:260] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.708163 1856 tablet_replica.cc:333] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.708587 1856 raft_consensus.cc:2243] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.708659 1206 ts_tablet_manager.cc:1434] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20251214 20:52:32.708814 1856 raft_consensus.cc:2272] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.709319 1856 ts_tablet_manager.cc:1916] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.710285 1202 tablet_bootstrap.cc:492] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.709018 1206 tablet_bootstrap.cc:492] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.714871 1206 tablet_bootstrap.cc:654] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.716827 1856 ts_tablet_manager.cc:1929] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.717921 1977 ts_tablet_manager.cc:1929] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.718029 1977 log.cc:1199] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/cbbfcef827ee4d548939aaf4783c8109
I20251214 20:52:32.718415 1977 ts_tablet_manager.cc:1950] T cbbfcef827ee4d548939aaf4783c8109 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.719298 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 452) took 3656 ms. Trace:
I20251214 20:52:32.719343 1202 tablet_bootstrap.cc:654] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.723172 1977 rpcz_store.cc:276] 1214 20:52:29.062492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.062556 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:32.669151 (+3606595us) ts_tablet_manager.cc:1151] Deleting tablet cbbfcef827ee4d548939aaf4783c8109
1214 20:52:32.669161 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.694462 (+ 25301us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.712602 (+ 18140us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.719275 (+ 6673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3532095}
I20251214 20:52:32.723589 1977 tablet_replica.cc:333] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.717710 1856 log.cc:1199] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/bc73d590531c409e9f60670435dc8223
I20251214 20:52:32.723795 1977 raft_consensus.cc:2243] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:32.724099 1977 pending_rounds.cc:70] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:32.724157 1977 pending_rounds.cc:77] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493773141839872 op_type: NO_OP noop_request { }
I20251214 20:52:32.724260 1977 raft_consensus.cc:2889] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:32.724349 1977 raft_consensus.cc:2272] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.724398 2777 raft_consensus.cc:493] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.724629 1856 ts_tablet_manager.cc:1950] T bc73d590531c409e9f60670435dc8223 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.724545 2777 raft_consensus.cc:515] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.724901 1977 ts_tablet_manager.cc:1916] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.725705 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feb19ea389af4ea6bb158a0e8aaeb094" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.726297 848 leader_election.cc:343] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.726576 2777 leader_election.cc:290] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:32.727478 851 leader_election.cc:343] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: feb19ea389af4ea6bb158a0e8aaeb094
I20251214 20:52:32.727640 851 leader_election.cc:304] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:32.728089 2777 raft_consensus.cc:2749] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.728335 1856 tablet_replica.cc:333] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.728729 1856 raft_consensus.cc:2243] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.729555 1856 raft_consensus.cc:2272] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.730134 1856 ts_tablet_manager.cc:1916] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.730489 1203 tablet_bootstrap.cc:654] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:32.741809 848 consensus_peers.cc:597] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:32.751276 1977 ts_tablet_manager.cc:1929] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.751391 1977 log.cc:1199] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/1dca369dc5f54a4ea1783e334c5670d6
I20251214 20:52:32.751688 1977 ts_tablet_manager.cc:1950] T 1dca369dc5f54a4ea1783e334c5670d6 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.752419 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 478) took 3179 ms. Trace:
I20251214 20:52:32.752496 1977 rpcz_store.cc:276] 1214 20:52:29.573069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.573369 (+ 300us) service_pool.cc:225] Handling call
1214 20:52:32.723534 (+3150165us) ts_tablet_manager.cc:1151] Deleting tablet 1dca369dc5f54a4ea1783e334c5670d6
1214 20:52:32.723550 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.748086 (+ 24536us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.751193 (+ 3107us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.752397 (+ 1204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3138870}
I20251214 20:52:32.752818 1977 tablet_replica.cc:333] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.752976 1977 raft_consensus.cc:2243] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:32.753260 1977 raft_consensus.cc:2272] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.753758 1977 ts_tablet_manager.cc:1916] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.754431 2889 raft_consensus.cc:493] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.754622 2889 raft_consensus.cc:515] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.755810 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a000951f61948ac85fdd431f3d162c2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.756189 2777 raft_consensus.cc:493] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.756320 2777 raft_consensus.cc:515] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:32.756426 848 leader_election.cc:343] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.756722 2889 leader_election.cc:290] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.756788 2777 leader_election.cc:290] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.757628 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7353593bda4fceaa389c393ddc19c6" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.758021 851 leader_election.cc:343] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 6a000951f61948ac85fdd431f3d162c2
I20251214 20:52:32.758174 851 leader_election.cc:304] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:32.758752 2777 raft_consensus.cc:2749] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:32.758038 848 leader_election.cc:343] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:32.759163 851 leader_election.cc:343] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: fc7353593bda4fceaa389c393ddc19c6
I20251214 20:52:32.759241 851 leader_election.cc:304] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:32.759321 1202 tablet_bootstrap.cc:492] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.759447 1202 ts_tablet_manager.cc:1403] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.016s
I20251214 20:52:32.759613 2777 raft_consensus.cc:2749] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.759764 2735 raft_consensus.cc:493] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.759897 2735 raft_consensus.cc:515] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.760188 1977 ts_tablet_manager.cc:1929] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.760166 1202 raft_consensus.cc:359] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.760286 1977 log.cc:1199] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/18e119bc44874aecb4e83c8d6040c8cd
I20251214 20:52:32.760311 1202 raft_consensus.cc:385] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.760358 1202 raft_consensus.cc:740] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.760462 2735 leader_election.cc:290] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.760561 1202 consensus_queue.cc:260] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.760663 1977 ts_tablet_manager.cc:1950] T 18e119bc44874aecb4e83c8d6040c8cd P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.761020 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a57409d4f54f17be792e6f3f17d7b7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.761106 1202 ts_tablet_manager.cc:1434] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:32.761401 1202 tablet_bootstrap.cc:492] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.761392 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a57409d4f54f17be792e6f3f17d7b7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.761458 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 481) took 3005 ms. Trace:
I20251214 20:52:32.761524 1977 rpcz_store.cc:276] 1214 20:52:29.756455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.758003 (+ 1548us) service_pool.cc:225] Handling call
1214 20:52:32.752754 (+2994751us) ts_tablet_manager.cc:1151] Deleting tablet 18e119bc44874aecb4e83c8d6040c8cd
1214 20:52:32.752764 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.756835 (+ 4071us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.760073 (+ 3238us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.761440 (+ 1367us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2994480}
W20251214 20:52:32.761862 989 leader_election.cc:343] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:32.762173 991 leader_election.cc:343] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.762253 991 leader_election.cc:304] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.762265 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 482) took 3005 ms. Trace:
I20251214 20:52:32.762369 1977 rpcz_store.cc:276] 1214 20:52:29.756555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.756965 (+ 410us) service_pool.cc:225] Handling call
1214 20:52:32.761780 (+3004815us) ts_tablet_manager.cc:1151] Deleting tablet d2a57409d4f54f17be792e6f3f17d7b7
1214 20:52:32.762074 (+ 294us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.762237 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2995183}
I20251214 20:52:32.762885 2735 raft_consensus.cc:2749] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.762998 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 486) took 2979 ms. Trace:
I20251214 20:52:32.763068 1977 rpcz_store.cc:276] 1214 20:52:29.783480 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.784151 (+ 671us) service_pool.cc:225] Handling call
1214 20:52:32.762638 (+2978487us) ts_tablet_manager.cc:1151] Deleting tablet f7f06eded6c340c58101fcc644ebcf50
1214 20:52:32.762857 (+ 219us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.762980 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2978262}
I20251214 20:52:32.764020 1982 ts_tablet_manager.cc:1929] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.766098 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 485) took 2982 ms. Trace:
I20251214 20:52:32.766609 1982 log.cc:1199] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/cbbfcef827ee4d548939aaf4783c8109
I20251214 20:52:32.767096 1982 ts_tablet_manager.cc:1950] T cbbfcef827ee4d548939aaf4783c8109 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.768044 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 441) took 3703 ms. Trace:
I20251214 20:52:32.768154 1982 rpcz_store.cc:276] 1214 20:52:29.064906 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.067024 (+ 2118us) service_pool.cc:225] Handling call
1214 20:52:32.634154 (+3567130us) ts_tablet_manager.cc:1151] Deleting tablet cbbfcef827ee4d548939aaf4783c8109
1214 20:52:32.634169 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.712396 (+ 78227us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.763849 (+ 51453us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.768021 (+ 4172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3566861}
I20251214 20:52:32.768579 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 469) took 3062 ms. Trace:
I20251214 20:52:32.768654 1982 rpcz_store.cc:276] 1214 20:52:29.706434 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.707209 (+ 775us) service_pool.cc:225] Handling call
1214 20:52:32.768458 (+3061249us) ts_tablet_manager.cc:1151] Deleting tablet f98be57ce1b14f20a3afcb79979b6c22
1214 20:52:32.768468 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.768564 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3060991}
I20251214 20:52:32.768958 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 467) took 3062 ms. Trace:
I20251214 20:52:32.769025 1982 rpcz_store.cc:276] 1214 20:52:29.706269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.711210 (+ 4941us) service_pool.cc:225] Handling call
1214 20:52:32.768862 (+3057652us) ts_tablet_manager.cc:1151] Deleting tablet 8e11723c042a4637839f1b13bed0935a
1214 20:52:32.768870 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.768946 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3057429}
I20251214 20:52:32.769428 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 468) took 3063 ms. Trace:
I20251214 20:52:32.769515 1982 rpcz_store.cc:276] 1214 20:52:29.706344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.711466 (+ 5122us) service_pool.cc:225] Handling call
1214 20:52:32.769321 (+3057855us) ts_tablet_manager.cc:1151] Deleting tablet 0b3e0a3bab7a47cb8b26f2094f5a1680
1214 20:52:32.769331 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.769410 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3057686}
I20251214 20:52:32.769831 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 470) took 3058 ms. Trace:
I20251214 20:52:32.769905 1982 rpcz_store.cc:276] 1214 20:52:29.711526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.711648 (+ 122us) service_pool.cc:225] Handling call
1214 20:52:32.769738 (+3058090us) ts_tablet_manager.cc:1151] Deleting tablet f7f06eded6c340c58101fcc644ebcf50
1214 20:52:32.769747 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.769816 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3057942}
I20251214 20:52:32.770157 1982 tablet_replica.cc:333] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.766217 1977 rpcz_store.cc:276] 1214 20:52:29.783399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.787319 (+ 3920us) service_pool.cc:225] Handling call
1214 20:52:32.765961 (+2978642us) ts_tablet_manager.cc:1151] Deleting tablet f98be57ce1b14f20a3afcb79979b6c22
1214 20:52:32.765975 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.766077 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2975837}
I20251214 20:52:32.770378 1982 raft_consensus.cc:2243] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.770444 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 484) took 2987 ms. Trace:
I20251214 20:52:32.770557 1982 raft_consensus.cc:2272] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.770516 1977 rpcz_store.cc:276] 1214 20:52:29.783288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.785059 (+ 1771us) service_pool.cc:225] Handling call
1214 20:52:32.770324 (+2985265us) ts_tablet_manager.cc:1151] Deleting tablet 0b3e0a3bab7a47cb8b26f2094f5a1680
1214 20:52:32.770334 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.770427 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2980158}
I20251214 20:52:32.771028 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 483) took 2997 ms. Trace:
I20251214 20:52:32.771106 1977 rpcz_store.cc:276] 1214 20:52:29.773344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.774120 (+ 776us) service_pool.cc:225] Handling call
1214 20:52:32.770934 (+2996814us) ts_tablet_manager.cc:1151] Deleting tablet 8e11723c042a4637839f1b13bed0935a
1214 20:52:32.770942 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.771012 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2950680}
I20251214 20:52:32.771378 1977 tablet_replica.cc:333] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.771575 1977 raft_consensus.cc:2243] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.771737 1977 raft_consensus.cc:2272] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.772014 2777 raft_consensus.cc:493] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.772179 2777 raft_consensus.cc:515] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.772291 1977 ts_tablet_manager.cc:1916] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.772789 2777 leader_election.cc:290] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.773458 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c1893b3761c49ceb7cd71b3bd9f2cb4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.773761 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c1893b3761c49ceb7cd71b3bd9f2cb4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:32.773937 1050 raft_consensus.cc:2468] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:32.773975 848 leader_election.cc:343] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.774369 851 leader_election.cc:304] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.774762 2777 raft_consensus.cc:2804] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:32.774830 2777 raft_consensus.cc:493] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:32.774886 2777 raft_consensus.cc:3060] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.775533 1202 tablet_bootstrap.cc:654] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.777892 2777 raft_consensus.cc:515] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.778908 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c1893b3761c49ceb7cd71b3bd9f2cb4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:32.779078 1050 raft_consensus.cc:3060] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.780791 1982 ts_tablet_manager.cc:1916] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.781000 2777 leader_election.cc:290] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.781143 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c1893b3761c49ceb7cd71b3bd9f2cb4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:32.781651 848 leader_election.cc:343] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.782577 1050 raft_consensus.cc:2468] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:32.782932 851 leader_election.cc:304] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.783393 2777 raft_consensus.cc:2804] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:32.783592 2777 raft_consensus.cc:697] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:32.783872 2777 consensus_queue.cc:237] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.785809 1206 tablet_bootstrap.cc:492] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.785935 1206 ts_tablet_manager.cc:1403] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.002s
I20251214 20:52:32.786549 1206 raft_consensus.cc:359] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.786710 1206 raft_consensus.cc:385] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.786751 1206 raft_consensus.cc:740] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.786913 1206 consensus_queue.cc:260] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.791378 1206 ts_tablet_manager.cc:1434] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251214 20:52:32.791654 1206 tablet_bootstrap.cc:492] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.793628 1206 tablet_bootstrap.cc:654] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.796109 1202 tablet_bootstrap.cc:492] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.796254 1202 ts_tablet_manager.cc:1403] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.001s
I20251214 20:52:32.797006 1202 raft_consensus.cc:359] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.797156 1202 raft_consensus.cc:385] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.797201 1202 raft_consensus.cc:740] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.797418 1202 consensus_queue.cc:260] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.797912 1202 ts_tablet_manager.cc:1434] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:32.798183 1202 tablet_bootstrap.cc:492] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.779086 1977 ts_tablet_manager.cc:1929] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:32.799327 1977 log.cc:1199] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/bf37c50e511f454ca4887c7f6ca1a746
I20251214 20:52:32.799777 1977 ts_tablet_manager.cc:1950] T bf37c50e511f454ca4887c7f6ca1a746 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.800086 1202 tablet_bootstrap.cc:654] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.800671 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 500) took 2475 ms. Trace:
I20251214 20:52:32.800769 1977 rpcz_store.cc:276] 1214 20:52:30.325064 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.325119 (+ 55us) service_pool.cc:225] Handling call
1214 20:52:32.771337 (+2446218us) ts_tablet_manager.cc:1151] Deleting tablet bf37c50e511f454ca4887c7f6ca1a746
1214 20:52:32.771345 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.775625 (+ 4280us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.778999 (+ 3374us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.800648 (+ 21649us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2307478}
I20251214 20:52:32.801122 1977 tablet_replica.cc:333] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.804422 1982 ts_tablet_manager.cc:1929] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.808410 1203 tablet_bootstrap.cc:492] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.808578 1203 ts_tablet_manager.cc:1403] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.009s
I20251214 20:52:32.808825 1977 raft_consensus.cc:2243] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:32.809296 1977 pending_rounds.cc:70] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:32.809387 1977 pending_rounds.cc:77] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493771360362496 op_type: NO_OP noop_request { }
I20251214 20:52:32.809453 1203 raft_consensus.cc:359] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.809634 1203 raft_consensus.cc:385] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.809574 1977 raft_consensus.cc:2889] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:32.809696 1203 raft_consensus.cc:740] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.809818 1977 raft_consensus.cc:2272] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.809913 1203 consensus_queue.cc:260] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.810582 1977 ts_tablet_manager.cc:1916] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.810783 1982 log.cc:1199] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/57f2314d33fa474ca36dafe0a0eae5ae
I20251214 20:52:32.813541 1982 ts_tablet_manager.cc:1950] T 57f2314d33fa474ca36dafe0a0eae5ae P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.816954 1203 ts_tablet_manager.cc:1434] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251214 20:52:32.818197 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 471) took 3078 ms. Trace:
I20251214 20:52:32.818369 1982 rpcz_store.cc:276] 1214 20:52:29.740218 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.740289 (+ 71us) service_pool.cc:225] Handling call
1214 20:52:32.770111 (+3029822us) ts_tablet_manager.cc:1151] Deleting tablet 57f2314d33fa474ca36dafe0a0eae5ae
1214 20:52:32.770119 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.786896 (+ 16777us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.804282 (+ 17386us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.818157 (+ 13875us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3029608}
I20251214 20:52:32.819345 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 466) took 3113 ms. Trace:
I20251214 20:52:32.819676 1982 rpcz_store.cc:276] 1214 20:52:29.706182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.747798 (+ 41616us) service_pool.cc:225] Handling call
1214 20:52:32.819198 (+3071400us) ts_tablet_manager.cc:1151] Deleting tablet d2a57409d4f54f17be792e6f3f17d7b7
1214 20:52:32.819210 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.819327 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3071173}
I20251214 20:52:32.823748 1982 tablet_replica.cc:333] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.824020 1982 raft_consensus.cc:2243] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.824071 1203 tablet_bootstrap.cc:492] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.824342 1982 raft_consensus.cc:2272] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.825029 1982 ts_tablet_manager.cc:1916] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.825655 1203 tablet_bootstrap.cc:654] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.828529 1202 tablet_bootstrap.cc:492] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.828701 1202 ts_tablet_manager.cc:1403] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.000s
I20251214 20:52:32.829407 1202 raft_consensus.cc:359] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.829589 1202 raft_consensus.cc:385] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.829644 1202 raft_consensus.cc:740] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.831364 1977 ts_tablet_manager.cc:1929] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.834895 1856 ts_tablet_manager.cc:1929] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.835103 1856 log.cc:1199] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/e9f6da5d29294f08a86c96c6821dda10
I20251214 20:52:32.835510 1856 ts_tablet_manager.cc:1950] T e9f6da5d29294f08a86c96c6821dda10 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.835877 1977 log.cc:1199] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/3d949abef1cb41a9905482d81a98d2c1
I20251214 20:52:32.836293 1977 ts_tablet_manager.cc:1950] T 3d949abef1cb41a9905482d81a98d2c1 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.836728 1856 tablet_replica.cc:333] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.836961 1856 raft_consensus.cc:2243] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.837162 1856 raft_consensus.cc:2272] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.837710 1856 ts_tablet_manager.cc:1916] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.839397 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 514) took 2500 ms. Trace:
I20251214 20:52:32.839592 1982 ts_tablet_manager.cc:1929] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.839514 1977 rpcz_store.cc:276] 1214 20:52:30.338927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.490946 (+152019us) service_pool.cc:225] Handling call
1214 20:52:32.801070 (+2310124us) ts_tablet_manager.cc:1151] Deleting tablet 3d949abef1cb41a9905482d81a98d2c1
1214 20:52:32.801080 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.816899 (+ 15819us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.831217 (+ 14318us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.839372 (+ 8155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2309938}
I20251214 20:52:32.839895 1982 log.cc:1199] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/18e119bc44874aecb4e83c8d6040c8cd
I20251214 20:52:32.839998 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 515) took 2501 ms. Trace:
I20251214 20:52:32.840071 1977 rpcz_store.cc:276] 1214 20:52:30.338988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.491182 (+152194us) service_pool.cc:225] Handling call
1214 20:52:32.839846 (+2348664us) ts_tablet_manager.cc:1151] Deleting tablet 18e95ff0e0da4c7680f594ba31cdc3a1
1214 20:52:32.839855 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.839982 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2345876}
I20251214 20:52:32.840380 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 516) took 2501 ms. Trace:
I20251214 20:52:32.840452 1977 rpcz_store.cc:276] 1214 20:52:30.339044 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.493991 (+154947us) service_pool.cc:225] Handling call
1214 20:52:32.840284 (+2346293us) ts_tablet_manager.cc:1151] Deleting tablet c249c096a41f4b4093eaf9afc6361249
1214 20:52:32.840292 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.840364 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346107}
I20251214 20:52:32.840548 1982 ts_tablet_manager.cc:1950] T 18e119bc44874aecb4e83c8d6040c8cd P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.840744 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 517) took 2501 ms. Trace:
I20251214 20:52:32.840811 1977 rpcz_store.cc:276] 1214 20:52:30.339105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.494195 (+155090us) service_pool.cc:225] Handling call
1214 20:52:32.840654 (+2346459us) ts_tablet_manager.cc:1151] Deleting tablet a6df388bdfbc40c183c4b14af2336a6a
1214 20:52:32.840662 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.840729 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346279}
I20251214 20:52:32.841131 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 519) took 2501 ms. Trace:
I20251214 20:52:32.841203 1977 rpcz_store.cc:276] 1214 20:52:30.339259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.510050 (+170791us) service_pool.cc:225] Handling call
1214 20:52:32.841043 (+2330993us) ts_tablet_manager.cc:1151] Deleting tablet 44c61824bacd406280ba03357f9f5b1d
1214 20:52:32.841050 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.841117 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2329677}
I20251214 20:52:32.841480 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 520) took 2342 ms. Trace:
I20251214 20:52:32.841548 1977 rpcz_store.cc:276] 1214 20:52:30.499633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.511387 (+ 11754us) service_pool.cc:225] Handling call
1214 20:52:32.841393 (+2330006us) ts_tablet_manager.cc:1151] Deleting tablet cbefd8f111084785bafd7e74ac443fd0
1214 20:52:32.841401 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.841465 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2327754}
I20251214 20:52:32.841842 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 521) took 2302 ms. Trace:
I20251214 20:52:32.841977 1977 rpcz_store.cc:276] 1214 20:52:30.539750 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.540067 (+ 317us) service_pool.cc:225] Handling call
1214 20:52:32.841757 (+2301690us) ts_tablet_manager.cc:1151] Deleting tablet bd1161eebe884fe6b347bf517269c047
1214 20:52:32.841765 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.841828 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2299446}
I20251214 20:52:32.829882 1202 consensus_queue.cc:260] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.841933 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 465) took 3135 ms. Trace:
I20251214 20:52:32.842216 1977 tablet_replica.cc:333] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.842285 1982 rpcz_store.cc:276] 1214 20:52:29.706068 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:29.789354 (+ 83286us) service_pool.cc:225] Handling call
1214 20:52:32.823687 (+3034333us) ts_tablet_manager.cc:1151] Deleting tablet 18e119bc44874aecb4e83c8d6040c8cd
1214 20:52:32.823699 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.836364 (+ 12665us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.839465 (+ 3101us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.841911 (+ 2446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3033876}
I20251214 20:52:32.842427 1977 raft_consensus.cc:2243] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.842588 1977 raft_consensus.cc:2272] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.842617 1202 ts_tablet_manager.cc:1434] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251214 20:52:32.842916 1202 tablet_bootstrap.cc:492] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.843209 1977 ts_tablet_manager.cc:1916] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.844393 1202 tablet_bootstrap.cc:654] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.842891 1982 tablet_replica.cc:333] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.844732 1982 raft_consensus.cc:2243] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.844970 1982 raft_consensus.cc:2272] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.845575 1982 ts_tablet_manager.cc:1916] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.845613 2735 raft_consensus.cc:493] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.845767 2735 raft_consensus.cc:515] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.846107 1203 tablet_bootstrap.cc:492] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.846235 1203 ts_tablet_manager.cc:1403] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20251214 20:52:32.846467 2735 leader_election.cc:290] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.846953 1203 raft_consensus.cc:359] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.847211 1203 raft_consensus.cc:385] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.847270 1203 raft_consensus.cc:740] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.847311 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6eca34627bbc4a3f9915be72f3128370" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.847468 1203 consensus_queue.cc:260] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.847904 1203 ts_tablet_manager.cc:1434] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:32.847935 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6eca34627bbc4a3f9915be72f3128370" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:32.848603 991 leader_election.cc:343] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:32.849056 989 leader_election.cc:343] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.849159 989 leader_election.cc:304] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.849627 2735 raft_consensus.cc:2749] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.850807 1206 tablet_bootstrap.cc:492] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.850942 1206 ts_tablet_manager.cc:1403] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.003s
I20251214 20:52:32.851604 1206 raft_consensus.cc:359] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.851755 1206 raft_consensus.cc:385] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.851797 1206 raft_consensus.cc:740] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.851989 1206 consensus_queue.cc:260] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.854238 1206 ts_tablet_manager.cc:1434] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:32.854511 1206 tablet_bootstrap.cc:492] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.857663 1203 tablet_bootstrap.cc:492] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.859649 1206 tablet_bootstrap.cc:654] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.864163 1202 tablet_bootstrap.cc:492] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.864357 1202 ts_tablet_manager.cc:1403] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20251214 20:52:32.865108 1202 raft_consensus.cc:359] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.865293 1202 raft_consensus.cc:385] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.865343 1202 raft_consensus.cc:740] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.867018 1203 tablet_bootstrap.cc:654] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.865553 1202 consensus_queue.cc:260] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.870610 1202 ts_tablet_manager.cc:1434] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251214 20:52:32.871515 1982 ts_tablet_manager.cc:1929] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251214 20:52:32.871647 1982 log.cc:1199] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/bf37c50e511f454ca4887c7f6ca1a746
I20251214 20:52:32.872017 1982 ts_tablet_manager.cc:1950] T bf37c50e511f454ca4887c7f6ca1a746 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.872864 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 485) took 2869 ms. Trace:
I20251214 20:52:32.872861 1856 ts_tablet_manager.cc:1929] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.872987 1856 log.cc:1199] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/f1eb4c3aa654480caaa69dbd5047365c
I20251214 20:52:32.872956 1982 rpcz_store.cc:276] 1214 20:52:30.002916 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.003001 (+ 85us) service_pool.cc:225] Handling call
1214 20:52:32.842841 (+2839840us) ts_tablet_manager.cc:1151] Deleting tablet bf37c50e511f454ca4887c7f6ca1a746
1214 20:52:32.842852 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.849257 (+ 6405us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.871376 (+ 22119us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.872840 (+ 1464us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2825770}
I20251214 20:52:32.873260 1856 ts_tablet_manager.cc:1950] T f1eb4c3aa654480caaa69dbd5047365c P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.873636 1202 tablet_bootstrap.cc:492] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.874341 1856 tablet_replica.cc:333] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.874539 1856 raft_consensus.cc:2243] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.874670 1856 raft_consensus.cc:2272] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.875223 1856 ts_tablet_manager.cc:1916] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.875334 1202 tablet_bootstrap.cc:654] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.878661 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 499) took 2603 ms. Trace:
I20251214 20:52:32.878786 1982 rpcz_store.cc:276] 1214 20:52:30.275697 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.276364 (+ 667us) service_pool.cc:225] Handling call
1214 20:52:32.878440 (+2602076us) ts_tablet_manager.cc:1151] Deleting tablet 18e95ff0e0da4c7680f594ba31cdc3a1
1214 20:52:32.878458 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.878619 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2601902}
I20251214 20:52:32.879603 1982 tablet_replica.cc:333] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.879832 1982 raft_consensus.cc:2243] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.880018 1982 raft_consensus.cc:2272] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.881168 1982 ts_tablet_manager.cc:1916] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.881376 1856 ts_tablet_manager.cc:1929] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.881445 2777 raft_consensus.cc:493] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.881498 1856 log.cc:1199] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/a0976ffcc649461fac74e20b5e402275
I20251214 20:52:32.881569 2777 raft_consensus.cc:515] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.881764 1856 ts_tablet_manager.cc:1950] T a0976ffcc649461fac74e20b5e402275 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.882205 2777 leader_election.cc:290] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.882774 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274185a4115b458e8e9a376bdbcc870b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.883385 848 leader_election.cc:343] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:32.883847 851 leader_election.cc:343] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 274185a4115b458e8e9a376bdbcc870b
I20251214 20:52:32.883953 851 leader_election.cc:304] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:32.884658 2777 raft_consensus.cc:2749] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.884981 1202 tablet_bootstrap.cc:492] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.885141 1202 ts_tablet_manager.cc:1403] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20251214 20:52:32.885711 1856 tablet_replica.cc:333] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.885947 1856 raft_consensus.cc:2243] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.885867 1202 raft_consensus.cc:359] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.886026 1202 raft_consensus.cc:385] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.886080 1202 raft_consensus.cc:740] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.886299 1856 raft_consensus.cc:2272] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.886293 1202 consensus_queue.cc:260] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.886799 1202 ts_tablet_manager.cc:1434] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:32.888054 1206 tablet_bootstrap.cc:492] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.888222 1206 ts_tablet_manager.cc:1403] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20251214 20:52:32.888927 1206 raft_consensus.cc:359] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.889184 1206 raft_consensus.cc:385] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.889245 1206 raft_consensus.cc:740] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.889478 1206 consensus_queue.cc:260] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.893054 1982 ts_tablet_manager.cc:1929] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.893072 1856 ts_tablet_manager.cc:1916] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.893219 1982 log.cc:1199] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/17550eb0ee6b476f8a8c304451f678b2
I20251214 20:52:32.893541 1982 ts_tablet_manager.cc:1950] T 17550eb0ee6b476f8a8c304451f678b2 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.894377 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 502) took 2618 ms. Trace:
I20251214 20:52:32.894481 1982 rpcz_store.cc:276] 1214 20:52:30.275980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.276545 (+ 565us) service_pool.cc:225] Handling call
1214 20:52:32.879551 (+2603006us) ts_tablet_manager.cc:1151] Deleting tablet 17550eb0ee6b476f8a8c304451f678b2
1214 20:52:32.879560 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.889828 (+ 10268us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.892919 (+ 3091us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.894356 (+ 1437us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2602903}
I20251214 20:52:32.894685 1206 ts_tablet_manager.cc:1434] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251214 20:52:32.894840 1982 tablet_replica.cc:333] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.895041 1982 raft_consensus.cc:2243] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.895525 1202 tablet_bootstrap.cc:492] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.897202 1202 tablet_bootstrap.cc:654] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.902199 1982 raft_consensus.cc:2272] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.903570 1982 ts_tablet_manager.cc:1916] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.908253 2884 raft_consensus.cc:493] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.908396 2884 raft_consensus.cc:515] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.909310 2884 leader_election.cc:290] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.910650 1982 ts_tablet_manager.cc:1929] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.910743 1982 log.cc:1199] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/96dc4bdabb234c3fa4c26bd1633e783d
I20251214 20:52:32.911031 1982 ts_tablet_manager.cc:1950] T 96dc4bdabb234c3fa4c26bd1633e783d P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.911718 1206 tablet_bootstrap.cc:492] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:32.911860 712 leader_election.cc:343] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 25dcf491653147e7831217c1ec7a2e0c
W20251214 20:52:32.912472 713 leader_election.cc:343] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 25dcf491653147e7831217c1ec7a2e0c
I20251214 20:52:32.912572 713 leader_election.cc:304] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.913007 2819 raft_consensus.cc:493] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.913152 1856 ts_tablet_manager.cc:1929] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.913129 2819 raft_consensus.cc:515] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.913247 1203 tablet_bootstrap.cc:492] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.913389 1203 ts_tablet_manager.cc:1403] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20251214 20:52:32.913774 2819 leader_election.cc:290] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.913908 2819 raft_consensus.cc:2749] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.914016 1203 raft_consensus.cc:359] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.914189 1203 raft_consensus.cc:385] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.914237 1203 raft_consensus.cc:740] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.914430 1203 consensus_queue.cc:260] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.914870 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e90d11d69b8d4fd49150aa67a8190e5b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:32.916646 713 leader_election.cc:343] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: e90d11d69b8d4fd49150aa67a8190e5b
I20251214 20:52:32.916744 713 leader_election.cc:304] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.917215 2884 raft_consensus.cc:2749] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.917506 1206 tablet_bootstrap.cc:654] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.918598 1203 ts_tablet_manager.cc:1434] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251214 20:52:32.918848 1203 tablet_bootstrap.cc:492] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.920147 1203 tablet_bootstrap.cc:654] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.913254 1856 log.cc:1199] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/ae948ffc81764692b92045f5befe0184
I20251214 20:52:32.921029 1856 ts_tablet_manager.cc:1950] T ae948ffc81764692b92045f5befe0184 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.925413 1856 tablet_replica.cc:333] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.925625 1856 raft_consensus.cc:2243] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.925782 1856 raft_consensus.cc:2272] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.926326 1856 ts_tablet_manager.cc:1916] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.926522 1977 ts_tablet_manager.cc:1929] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.926708 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 503) took 2650 ms. Trace:
I20251214 20:52:32.926810 1982 rpcz_store.cc:276] 1214 20:52:30.276048 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.276657 (+ 609us) service_pool.cc:225] Handling call
1214 20:52:32.894787 (+2618130us) ts_tablet_manager.cc:1151] Deleting tablet 96dc4bdabb234c3fa4c26bd1633e783d
1214 20:52:32.894797 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.907739 (+ 12942us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.910563 (+ 2824us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.926681 (+ 16118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2618037}
I20251214 20:52:32.927296 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 500) took 2651 ms. Trace:
I20251214 20:52:32.927189 1977 log.cc:1199] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/96dc4bdabb234c3fa4c26bd1633e783d
I20251214 20:52:32.927372 1982 rpcz_store.cc:276] 1214 20:52:30.275818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.276772 (+ 954us) service_pool.cc:225] Handling call
1214 20:52:32.927138 (+2650366us) ts_tablet_manager.cc:1151] Deleting tablet c249c096a41f4b4093eaf9afc6361249
1214 20:52:32.927150 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.927281 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2650167}
I20251214 20:52:32.927685 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 504) took 2627 ms. Trace:
I20251214 20:52:32.927757 1982 rpcz_store.cc:276] 1214 20:52:30.300330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.308549 (+ 8219us) service_pool.cc:225] Handling call
1214 20:52:32.927590 (+2619041us) ts_tablet_manager.cc:1151] Deleting tablet 44c61824bacd406280ba03357f9f5b1d
1214 20:52:32.927598 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.927670 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2618816}
I20251214 20:52:32.928045 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 501) took 2652 ms. Trace:
I20251214 20:52:32.928112 1982 rpcz_store.cc:276] 1214 20:52:30.275905 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.276313 (+ 408us) service_pool.cc:225] Handling call
1214 20:52:32.927958 (+2651645us) ts_tablet_manager.cc:1151] Deleting tablet a6df388bdfbc40c183c4b14af2336a6a
1214 20:52:32.927966 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.928031 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2595111}
I20251214 20:52:32.928411 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 505) took 2497 ms. Trace:
I20251214 20:52:32.928488 1982 rpcz_store.cc:276] 1214 20:52:30.431312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.431377 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:32.928317 (+2496940us) ts_tablet_manager.cc:1151] Deleting tablet cbefd8f111084785bafd7e74ac443fd0
1214 20:52:32.928325 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.928395 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2496723}
I20251214 20:52:32.928768 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 506) took 2426 ms. Trace:
I20251214 20:52:32.928833 1982 rpcz_store.cc:276] 1214 20:52:30.502106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.502168 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:32.928682 (+2426514us) ts_tablet_manager.cc:1151] Deleting tablet bd1161eebe884fe6b347bf517269c047
1214 20:52:32.928690 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.928755 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2426300}
I20251214 20:52:32.928503 2777 raft_consensus.cc:493] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.929067 1982 tablet_replica.cc:333] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:32.929047 2777 raft_consensus.cc:515] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.929258 1982 raft_consensus.cc:2243] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.929418 1982 raft_consensus.cc:2272] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.929813 2777 leader_election.cc:290] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.930001 1982 ts_tablet_manager.cc:1916] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.930660 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1eb4c3aa654480caaa69dbd5047365c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:32.931219 848 leader_election.cc:343] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:32.932332 851 leader_election.cc:343] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: f1eb4c3aa654480caaa69dbd5047365c
I20251214 20:52:32.932439 851 leader_election.cc:304] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:32.932883 2777 raft_consensus.cc:2749] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.933239 2884 raft_consensus.cc:493] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.933355 2884 raft_consensus.cc:515] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.933841 1856 ts_tablet_manager.cc:1929] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.933939 1856 log.cc:1199] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/eb29fd8c85344ed8aa8d880e2ed240bc
I20251214 20:52:32.933918 2884 leader_election.cc:290] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:32.934196 1856 ts_tablet_manager.cc:1950] T eb29fd8c85344ed8aa8d880e2ed240bc P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:32.934985 1856 tablet_replica.cc:333] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:32.935248 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "876d2a1f1d1c4f6882a5857b43568ee5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:32.934592 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "876d2a1f1d1c4f6882a5857b43568ee5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:32.936540 712 leader_election.cc:304] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:32.937057 2884 raft_consensus.cc:2749] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:32.928472 1977 ts_tablet_manager.cc:1950] T 96dc4bdabb234c3fa4c26bd1633e783d P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:32.944576 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 518) took 2605 ms. Trace:
I20251214 20:52:32.947254 1982 ts_tablet_manager.cc:1929] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:32.947399 1982 log.cc:1199] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/e90d11d69b8d4fd49150aa67a8190e5b
I20251214 20:52:32.947723 1982 ts_tablet_manager.cc:1950] T e90d11d69b8d4fd49150aa67a8190e5b P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:32.948604 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 510) took 2234 ms. Trace:
I20251214 20:52:32.948694 1982 rpcz_store.cc:276] 1214 20:52:30.713849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.713919 (+ 70us) service_pool.cc:225] Handling call
1214 20:52:32.929021 (+2215102us) ts_tablet_manager.cc:1151] Deleting tablet e90d11d69b8d4fd49150aa67a8190e5b
1214 20:52:32.929029 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.933224 (+ 4195us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.946995 (+ 13771us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.948582 (+ 1587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884495}
I20251214 20:52:32.949296 1977 rpcz_store.cc:276] 1214 20:52:30.339197 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.494389 (+155192us) service_pool.cc:225] Handling call
1214 20:52:32.842172 (+2347783us) ts_tablet_manager.cc:1151] Deleting tablet 96dc4bdabb234c3fa4c26bd1633e783d
1214 20:52:32.842180 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.901139 (+ 58959us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.907915 (+ 6776us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.944542 (+ 36627us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2299764}
I20251214 20:52:32.949733 1977 tablet_replica.cc:333] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:32.949899 1202 tablet_bootstrap.cc:492] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:32.950035 1202 ts_tablet_manager.cc:1403] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.017s
I20251214 20:52:32.950127 1977 raft_consensus.cc:2243] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.950340 1977 raft_consensus.cc:2272] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.950932 1977 ts_tablet_manager.cc:1916] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.951225 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 528) took 2086 ms. Trace:
I20251214 20:52:32.950881 1202 raft_consensus.cc:359] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.951409 1202 raft_consensus.cc:385] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.951462 1202 raft_consensus.cc:740] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.951666 1202 consensus_queue.cc:260] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.952139 1202 ts_tablet_manager.cc:1434] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:32.952391 1202 tablet_bootstrap.cc:492] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:32.951318 1982 rpcz_store.cc:276] 1214 20:52:30.865281 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.144919 (+279638us) service_pool.cc:225] Handling call
1214 20:52:32.949139 (+1804220us) ts_tablet_manager.cc:1151] Deleting tablet 2f75657d7bc44f6393c55ab119eb7dc2
1214 20:52:32.949149 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.951202 (+ 2053us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1803962}
I20251214 20:52:32.953651 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 529) took 2075 ms. Trace:
I20251214 20:52:32.953785 1982 rpcz_store.cc:276] 1214 20:52:30.878118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.145184 (+267066us) service_pool.cc:225] Handling call
1214 20:52:32.953010 (+1807826us) ts_tablet_manager.cc:1151] Deleting tablet 348d8dcef97042279bc0d5ff31d3c137
1214 20:52:32.953043 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.953630 (+ 587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1807710}
I20251214 20:52:32.954468 1982 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57406 (request call id 530) took 2068 ms. Trace:
I20251214 20:52:32.954578 1982 rpcz_store.cc:276] 1214 20:52:30.885689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.145301 (+259612us) service_pool.cc:225] Handling call
1214 20:52:32.954288 (+1808987us) ts_tablet_manager.cc:1151] Deleting tablet 1def5f0de5ce4eb0bed36f626c4f0683
1214 20:52:32.954320 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.954450 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808900}
I20251214 20:52:32.964478 1202 tablet_bootstrap.cc:654] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:32.964576 1206 tablet_bootstrap.cc:492] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.964766 1206 ts_tablet_manager.cc:1403] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.007s
I20251214 20:52:32.965476 1206 raft_consensus.cc:359] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.965672 1206 raft_consensus.cc:385] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.965718 1206 raft_consensus.cc:740] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.965899 1206 consensus_queue.cc:260] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.966338 1206 ts_tablet_manager.cc:1434] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251214 20:52:32.966622 1206 tablet_bootstrap.cc:492] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:32.972863 1856 raft_consensus.cc:2243] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:32.973083 1856 raft_consensus.cc:2272] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:32.973712 1856 ts_tablet_manager.cc:1916] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:32.974591 1206 tablet_bootstrap.cc:654] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:32.977108 848 consensus_peers.cc:597] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:32.984555 1977 ts_tablet_manager.cc:1929] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:32.984736 1977 log.cc:1199] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/e90d11d69b8d4fd49150aa67a8190e5b
I20251214 20:52:32.986074 1203 tablet_bootstrap.cc:492] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:32.986253 1203 ts_tablet_manager.cc:1403] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.005s
I20251214 20:52:32.986989 1203 raft_consensus.cc:359] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.987207 1203 raft_consensus.cc:385] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.987258 1203 raft_consensus.cc:740] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.987494 1203 consensus_queue.cc:260] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:32.988545 1206 tablet_bootstrap.cc:492] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:32.988715 1206 ts_tablet_manager.cc:1403] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.002s
I20251214 20:52:32.988691 2777 raft_consensus.cc:493] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:32.988866 2777 raft_consensus.cc:515] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:32.989388 1206 raft_consensus.cc:359] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.989540 1206 raft_consensus.cc:385] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:32.989569 2777 leader_election.cc:290] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:32.989585 1206 raft_consensus.cc:740] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:32.989804 1206 consensus_queue.cc:260] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:32.990331 1203 ts_tablet_manager.cc:1434] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:32.990581 1203 tablet_bootstrap.cc:492] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:32.990775 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f11834cabdad445b8e53cd237d146282" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:32.991600 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f11834cabdad445b8e53cd237d146282" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:32.991825 1050 raft_consensus.cc:2468] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:32.991983 1203 tablet_bootstrap.cc:654] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:32.992714 848 leader_election.cc:343] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:32.995195 851 leader_election.cc:304] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:32.996006 2777 raft_consensus.cc:2804] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:32.996026 1206 ts_tablet_manager.cc:1434] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251214 20:52:32.996093 2777 raft_consensus.cc:493] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:32.996165 2777 raft_consensus.cc:3060] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:32.999079 2777 raft_consensus.cc:515] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.003355 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f11834cabdad445b8e53cd237d146282" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:33.004103 848 leader_election.cc:343] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.005039 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f11834cabdad445b8e53cd237d146282" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.005260 1050 raft_consensus.cc:3060] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.006327 1206 tablet_bootstrap.cc:492] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.008208 1206 tablet_bootstrap.cc:654] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.009362 1050 raft_consensus.cc:2468] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:33.009928 851 leader_election.cc:304] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.010304 2889 raft_consensus.cc:2804] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.010418 2889 raft_consensus.cc:697] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.010738 2889 consensus_queue.cc:237] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.012789 1856 ts_tablet_manager.cc:1929] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:33.012914 1856 log.cc:1199] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/fe24e4f4e74040249068505058c15622
I20251214 20:52:33.013175 1856 ts_tablet_manager.cc:1950] T fe24e4f4e74040249068505058c15622 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:33.013311 2777 leader_election.cc:290] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.014104 1856 tablet_replica.cc:333] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:33.014282 1856 raft_consensus.cc:2243] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:33.014407 1856 raft_consensus.cc:2272] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:33.014850 1856 ts_tablet_manager.cc:1916] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:33.015833 1203 tablet_bootstrap.cc:492] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.015964 1203 ts_tablet_manager.cc:1403] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20251214 20:52:33.016700 1203 raft_consensus.cc:359] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.016863 1203 raft_consensus.cc:385] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.016924 1203 raft_consensus.cc:740] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.017127 1203 consensus_queue.cc:260] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.017946 1206 tablet_bootstrap.cc:492] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.018132 1206 ts_tablet_manager.cc:1403] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20251214 20:52:33.018769 1977 ts_tablet_manager.cc:1950] T e90d11d69b8d4fd49150aa67a8190e5b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:33.018952 1206 raft_consensus.cc:359] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.019162 1206 raft_consensus.cc:385] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.019227 1206 raft_consensus.cc:740] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.019457 1206 consensus_queue.cc:260] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.020116 1206 ts_tablet_manager.cc:1434] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:33.020464 1206 tablet_bootstrap.cc:492] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.021458 2786 consensus_queue.cc:1048] T d424e3387698434f92e84eb1d43f5949 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:33.021642 1203 ts_tablet_manager.cc:1434] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251214 20:52:33.022238 1203 tablet_bootstrap.cc:492] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.023558 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 524) took 2274 ms. Trace:
I20251214 20:52:33.023792 1203 tablet_bootstrap.cc:654] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.024662 1977 rpcz_store.cc:276] 1214 20:52:30.749059 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:30.752146 (+ 3087us) service_pool.cc:225] Handling call
1214 20:52:32.949680 (+2197534us) ts_tablet_manager.cc:1151] Deleting tablet e90d11d69b8d4fd49150aa67a8190e5b
1214 20:52:32.949688 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:32.968192 (+ 18504us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:32.984377 (+ 16185us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.023528 (+ 39151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2197336}
I20251214 20:52:33.025553 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 544) took 2046 ms. Trace:
I20251214 20:52:33.027267 1977 rpcz_store.cc:276] 1214 20:52:30.978712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.040326 (+ 61614us) service_pool.cc:225] Handling call
1214 20:52:33.025381 (+1985055us) ts_tablet_manager.cc:1151] Deleting tablet 2f75657d7bc44f6393c55ab119eb7dc2
1214 20:52:33.025396 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.025534 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1982917}
I20251214 20:52:33.027702 1206 tablet_bootstrap.cc:654] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.028024 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 545) took 1459 ms. Trace:
I20251214 20:52:33.034525 2884 raft_consensus.cc:493] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.034682 2884 raft_consensus.cc:515] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.035302 2884 leader_election.cc:290] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.036087 2884 raft_consensus.cc:493] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.036266 2884 raft_consensus.cc:515] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.036914 2884 leader_election.cc:290] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:33.039443 712 leader_election.cc:343] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 453cf734f8b34b59aa246d618ffeade5
W20251214 20:52:33.039646 712 leader_election.cc:343] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 0c973e0111804b8b869068bdcc5869bf
W20251214 20:52:33.041376 713 leader_election.cc:343] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 453cf734f8b34b59aa246d618ffeade5
I20251214 20:52:33.041514 713 leader_election.cc:304] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.042021 2906 raft_consensus.cc:493] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.042145 2906 raft_consensus.cc:515] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.042716 2906 leader_election.cc:290] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.043267 2884 raft_consensus.cc:2749] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.044163 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292a28b12df246a8bc855d81c530746e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.044943 989 leader_election.cc:343] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.049036 713 leader_election.cc:343] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 0c973e0111804b8b869068bdcc5869bf
I20251214 20:52:33.043457 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292a28b12df246a8bc855d81c530746e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.049175 713 leader_election.cc:304] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.049599 2884 raft_consensus.cc:2749] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.049887 1202 tablet_bootstrap.cc:492] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.050035 1202 ts_tablet_manager.cc:1403] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.098s user 0.026s sys 0.017s
W20251214 20:52:33.050155 991 leader_election.cc:343] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.050256 991 leader_election.cc:304] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.050166 1977 rpcz_store.cc:276] 1214 20:52:31.568501 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.568564 (+ 63us) service_pool.cc:225] Handling call
1214 20:52:33.027892 (+1459328us) ts_tablet_manager.cc:1151] Deleting tablet 348d8dcef97042279bc0d5ff31d3c137
1214 20:52:33.027901 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.028004 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1459074}
I20251214 20:52:33.050596 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 546) took 1482 ms. Trace:
I20251214 20:52:33.050673 1977 rpcz_store.cc:276] 1214 20:52:31.568618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.568664 (+ 46us) service_pool.cc:225] Handling call
1214 20:52:33.050472 (+1481808us) ts_tablet_manager.cc:1151] Deleting tablet 1def5f0de5ce4eb0bed36f626c4f0683
1214 20:52:33.050482 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.050580 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":154,"tablet-delete.queue_time_us":1481654}
I20251214 20:52:33.050818 2906 raft_consensus.cc:2749] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.050782 1202 raft_consensus.cc:359] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.050938 1202 raft_consensus.cc:385] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.050958 1977 tablet_replica.cc:333] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:33.050997 1202 raft_consensus.cc:740] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.051219 1202 consensus_queue.cc:260] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.051682 1202 ts_tablet_manager.cc:1434] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:33.057688 1856 ts_tablet_manager.cc:1929] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:33.061600 2777 consensus_queue.cc:1048] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:33.063266 1856 log.cc:1199] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/088c72a6f5014b6fbc6ee060b6b7a590
I20251214 20:52:33.063886 1203 tablet_bootstrap.cc:492] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.064044 1203 ts_tablet_manager.cc:1403] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.001s
I20251214 20:52:33.064710 1203 raft_consensus.cc:359] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.064940 1203 raft_consensus.cc:385] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.065034 1203 raft_consensus.cc:740] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.065304 1203 consensus_queue.cc:260] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.065801 1203 ts_tablet_manager.cc:1434] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:33.066107 1203 tablet_bootstrap.cc:492] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.067458 1856 ts_tablet_manager.cc:1950] T 088c72a6f5014b6fbc6ee060b6b7a590 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:33.067878 1203 tablet_bootstrap.cc:654] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:33.072317 848 consensus_peers.cc:597] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:33.073161 1202 tablet_bootstrap.cc:492] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.074649 1856 tablet_replica.cc:333] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:33.074973 1856 raft_consensus.cc:2243] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:33.074997 1202 tablet_bootstrap.cc:654] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.075234 1856 raft_consensus.cc:2272] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:33.075790 1856 ts_tablet_manager.cc:1916] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:33.076192 1977 raft_consensus.cc:2243] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:33.076399 1977 raft_consensus.cc:2272] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:33.077026 1977 ts_tablet_manager.cc:1916] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:33.079824 2889 raft_consensus.cc:493] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.079981 2889 raft_consensus.cc:515] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.081259 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f945814952b14131ba4e0c67a403a680" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.081931 848 leader_election.cc:343] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.082913 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f945814952b14131ba4e0c67a403a680" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.083204 1056 raft_consensus.cc:2468] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:33.083660 851 leader_election.cc:304] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.083948 2777 raft_consensus.cc:2804] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.084025 2777 raft_consensus.cc:493] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.084082 2777 raft_consensus.cc:3060] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.086865 2777 raft_consensus.cc:515] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.087580 2884 raft_consensus.cc:493] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.087742 2884 raft_consensus.cc:515] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.087983 1203 tablet_bootstrap.cc:492] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.088140 1203 ts_tablet_manager.cc:1403] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20251214 20:52:33.088443 2884 leader_election.cc:290] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.088797 1203 raft_consensus.cc:359] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.088969 1203 raft_consensus.cc:385] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.089017 1203 raft_consensus.cc:740] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.089275 1203 consensus_queue.cc:260] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:33.089658 713 leader_election.cc:343] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 58ac06f852fc4abea5cb213926002ca1
I20251214 20:52:33.089843 1203 ts_tablet_manager.cc:1434] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251214 20:52:33.090025 712 leader_election.cc:343] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 58ac06f852fc4abea5cb213926002ca1
I20251214 20:52:33.090123 1203 tablet_bootstrap.cc:492] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.090118 712 leader_election.cc:304] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.090572 2777 leader_election.cc:290] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.090672 2884 raft_consensus.cc:2749] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.090840 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f945814952b14131ba4e0c67a403a680" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.091056 1056 raft_consensus.cc:3060] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.091773 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f945814952b14131ba4e0c67a403a680" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:33.092406 848 leader_election.cc:343] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.092734 1202 tablet_bootstrap.cc:492] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.092881 1202 ts_tablet_manager.cc:1403] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.007s
I20251214 20:52:33.093585 1202 raft_consensus.cc:359] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.093752 1202 raft_consensus.cc:385] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.093806 1202 raft_consensus.cc:740] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.093999 1202 consensus_queue.cc:260] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.095320 1202 ts_tablet_manager.cc:1434] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:33.095580 1202 tablet_bootstrap.cc:492] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.097040 1202 tablet_bootstrap.cc:654] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.091490 1203 tablet_bootstrap.cc:654] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.102113 2777 raft_consensus.cc:493] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.102270 2777 raft_consensus.cc:515] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.102877 2777 leader_election.cc:290] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.104069 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc73d590531c409e9f60670435dc8223" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.104591 851 leader_election.cc:343] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: bc73d590531c409e9f60670435dc8223
W20251214 20:52:33.104807 848 leader_election.cc:343] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.104893 848 leader_election.cc:304] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.105312 2777 raft_consensus.cc:2749] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.105753 1056 raft_consensus.cc:2468] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:33.106247 851 leader_election.cc:304] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.106622 2777 raft_consensus.cc:2804] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.106846 2777 raft_consensus.cc:697] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.107169 2777 consensus_queue.cc:237] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.108275 1856 ts_tablet_manager.cc:1929] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:33.108453 1856 log.cc:1199] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/3b47ca1a03944cf1847d3578856dafa7
I20251214 20:52:33.108786 1856 ts_tablet_manager.cc:1950] T 3b47ca1a03944cf1847d3578856dafa7 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:33.109177 2889 leader_election.cc:290] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.110586 1856 tablet_replica.cc:333] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:33.111166 1856 raft_consensus.cc:2243] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:33.111557 1856 raft_consensus.cc:2272] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:33.112357 1203 tablet_bootstrap.cc:492] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.113155 1203 ts_tablet_manager.cc:1403] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20251214 20:52:33.113806 1203 raft_consensus.cc:359] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.114006 1203 raft_consensus.cc:385] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.114104 1203 raft_consensus.cc:740] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.114334 1203 consensus_queue.cc:260] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.114921 1203 ts_tablet_manager.cc:1434] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:33.115511 1203 tablet_bootstrap.cc:492] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.116828 1203 tablet_bootstrap.cc:654] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.117609 1856 ts_tablet_manager.cc:1916] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:33.131896 2777 consensus_queue.cc:1048] T c6a8863ace4541aea7463af24db27142 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:33.133080 2906 raft_consensus.cc:493] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.133292 2906 raft_consensus.cc:515] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.133885 2906 leader_election.cc:290] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.135200 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e621f29c01448399d22615284fccd4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.135264 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e621f29c01448399d22615284fccd4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.136045 989 leader_election.cc:343] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.136487 1203 tablet_bootstrap.cc:492] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.136638 1203 ts_tablet_manager.cc:1403] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
W20251214 20:52:33.136680 991 leader_election.cc:343] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.136757 991 leader_election.cc:304] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.137153 2735 raft_consensus.cc:2749] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.137321 1203 raft_consensus.cc:359] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.137470 1203 raft_consensus.cc:385] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.137524 1203 raft_consensus.cc:740] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.137729 1203 consensus_queue.cc:260] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.138325 1203 ts_tablet_manager.cc:1434] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:33.138598 1203 tablet_bootstrap.cc:492] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.140152 1203 tablet_bootstrap.cc:654] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.144716 1856 ts_tablet_manager.cc:1929] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:33.144928 1856 log.cc:1199] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/d01db63fd5d2474a9defdce0b98fc29d
I20251214 20:52:33.145298 1856 ts_tablet_manager.cc:1950] T d01db63fd5d2474a9defdce0b98fc29d P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:33.145761 2884 raft_consensus.cc:493] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.145917 2884 raft_consensus.cc:515] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.146356 1856 tablet_replica.cc:333] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:33.146524 1856 raft_consensus.cc:2243] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:33.146569 2884 leader_election.cc:290] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.146652 1856 raft_consensus.cc:2272] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:33.147207 1856 ts_tablet_manager.cc:1916] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:33.147501 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6bdfbe4d15345de8be5de00bfe4ca07" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.148293 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6bdfbe4d15345de8be5de00bfe4ca07" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.148712 2735 raft_consensus.cc:493] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.148825 2735 raft_consensus.cc:515] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.148910 713 leader_election.cc:304] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.149374 2735 leader_election.cc:290] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.149461 2884 raft_consensus.cc:2749] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.150144 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7aa508b11944aa8e268741c1998a9e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.150485 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7aa508b11944aa8e268741c1998a9e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.150655 991 leader_election.cc:343] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.150943 989 leader_election.cc:343] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.151048 989 leader_election.cc:304] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.151506 2735 raft_consensus.cc:2749] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.154683 1203 tablet_bootstrap.cc:492] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.154815 1203 ts_tablet_manager.cc:1403] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20251214 20:52:33.155509 1203 raft_consensus.cc:359] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.155679 1203 raft_consensus.cc:385] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.155728 1203 raft_consensus.cc:740] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.155941 1203 consensus_queue.cc:260] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.156823 1203 ts_tablet_manager.cc:1434] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:33.157086 1203 tablet_bootstrap.cc:492] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.158406 1203 tablet_bootstrap.cc:654] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.175240 1206 tablet_bootstrap.cc:492] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.175635 1856 ts_tablet_manager.cc:1929] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:33.176297 1856 log.cc:1199] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/0d7c7b3f62764ff497f4d70b53eed7de
I20251214 20:52:33.176671 1856 ts_tablet_manager.cc:1950] T 0d7c7b3f62764ff497f4d70b53eed7de P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:33.177011 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 618) took 1007 ms. Trace:
I20251214 20:52:33.177160 1206 ts_tablet_manager.cc:1403] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.157s user 0.015s sys 0.033s
I20251214 20:52:33.177143 1856 rpcz_store.cc:276] 1214 20:52:32.169440 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:32.176212 (+ 6772us) service_pool.cc:225] Handling call
1214 20:52:33.146311 (+970099us) ts_tablet_manager.cc:1151] Deleting tablet 0d7c7b3f62764ff497f4d70b53eed7de
1214 20:52:33.146320 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.160346 (+ 14026us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.175517 (+ 15171us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.176993 (+ 1476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":969956}
I20251214 20:52:33.177728 1856 tablet_replica.cc:333] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:33.177995 1856 raft_consensus.cc:2243] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:33.178256 1856 raft_consensus.cc:2272] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:33.178750 1206 raft_consensus.cc:359] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.178968 1206 raft_consensus.cc:385] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.179061 1206 raft_consensus.cc:740] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.179342 1206 consensus_queue.cc:260] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.179612 1856 ts_tablet_manager.cc:1916] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:33.185913 1206 ts_tablet_manager.cc:1434] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251214 20:52:33.180402 1203 tablet_bootstrap.cc:492] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.186213 1203 ts_tablet_manager.cc:1403] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20251214 20:52:33.186297 1206 tablet_bootstrap.cc:492] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.186935 1203 raft_consensus.cc:359] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.187107 1203 raft_consensus.cc:385] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.187181 1203 raft_consensus.cc:740] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.187374 1203 consensus_queue.cc:260] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.187844 1203 ts_tablet_manager.cc:1434] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:33.188102 1203 tablet_bootstrap.cc:492] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.189646 1203 tablet_bootstrap.cc:654] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.192560 1206 tablet_bootstrap.cc:654] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.196312 1856 ts_tablet_manager.cc:1929] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:33.206527 1856 log.cc:1199] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/65e79db74fa046698507c83416ee1669
I20251214 20:52:33.207063 1856 ts_tablet_manager.cc:1950] T 65e79db74fa046698507c83416ee1669 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:33.208325 1856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40270 (request call id 627) took 1031 ms. Trace:
I20251214 20:52:33.208757 1856 rpcz_store.cc:276] 1214 20:52:32.176602 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:32.177924 (+ 1322us) service_pool.cc:225] Handling call
1214 20:52:33.177679 (+999755us) ts_tablet_manager.cc:1151] Deleting tablet 65e79db74fa046698507c83416ee1669
1214 20:52:33.177688 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.193140 (+ 15452us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.196211 (+ 3071us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.208302 (+ 12091us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":999618}
I20251214 20:52:33.209353 1203 tablet_bootstrap.cc:492] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.209492 1203 ts_tablet_manager.cc:1403] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20251214 20:52:33.210126 1203 raft_consensus.cc:359] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.210283 1203 raft_consensus.cc:385] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.210331 1203 raft_consensus.cc:740] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.211467 1203 consensus_queue.cc:260] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.211930 1203 ts_tablet_manager.cc:1434] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:33.212196 1203 tablet_bootstrap.cc:492] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.218751 2889 raft_consensus.cc:493] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.219034 2889 raft_consensus.cc:515] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.220582 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb29fd8c85344ed8aa8d880e2ed240bc" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.221446 848 leader_election.cc:343] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.219170 2936 raft_consensus.cc:493] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.223634 2936 raft_consensus.cc:515] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:33.224463 1203 tablet_bootstrap.cc:654] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:33.224524 851 leader_election.cc:343] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: eb29fd8c85344ed8aa8d880e2ed240bc
I20251214 20:52:33.224639 851 leader_election.cc:304] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.224946 2786 raft_consensus.cc:2749] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.225077 2936 leader_election.cc:290] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:33.228199 2889 leader_election.cc:290] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.239584 2937 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.239840 2937 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:33.241144 2397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 7 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
I20251214 20:52:33.241396 2397 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c48b8eaf30c4e2caa94a51d087406d4 in term 6.
I20251214 20:52:33.241465 2889 raft_consensus.cc:493] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.241643 2889 raft_consensus.cc:515] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.241809 1334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" is_pre_election: true
I20251214 20:52:33.242015 1334 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6713a20a82b434f8a8bc122113e26ad in term 6.
I20251214 20:52:33.242815 1297 leader_election.cc:304] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, e6713a20a82b434f8a8bc122113e26ad; no voters:
I20251214 20:52:33.242898 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "082ec70b8ee646b898d9201ad073d292" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.242920 2291 leader_election.cc:304] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, e6713a20a82b434f8a8bc122113e26ad; no voters:
I20251214 20:52:33.243309 2936 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251214 20:52:33.243402 2936 raft_consensus.cc:493] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.243474 2936 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 6 FOLLOWER]: Advancing to term 7
W20251214 20:52:33.243925 848 leader_election.cc:343] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.244838 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "082ec70b8ee646b898d9201ad073d292" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.245101 1050 raft_consensus.cc:2468] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:33.245486 851 leader_election.cc:304] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.245880 2777 raft_consensus.cc:2804] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.245955 2777 raft_consensus.cc:493] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.246032 2777 raft_consensus.cc:3060] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.246997 2936 raft_consensus.cc:515] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:33.247925 1203 tablet_bootstrap.cc:492] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.248090 1203 ts_tablet_manager.cc:1403] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.004s
I20251214 20:52:33.248626 2936 leader_election.cc:290] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 7 election: Requested vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), 7c48b8eaf30c4e2caa94a51d087406d4 (127.0.104.60:35099)
I20251214 20:52:33.248840 1203 raft_consensus.cc:359] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.248999 1203 raft_consensus.cc:385] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.249068 1203 raft_consensus.cc:740] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.249193 2777 raft_consensus.cc:515] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.249266 1203 consensus_queue.cc:260] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.249647 1334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e6713a20a82b434f8a8bc122113e26ad" candidate_term: 7 candidate_status { last_received { term: 4 index: 955 } } ignore_live_leader: false dest_uuid: "7c48b8eaf30c4e2caa94a51d087406d4"
I20251214 20:52:33.249879 1203 ts_tablet_manager.cc:1434] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:33.249851 1334 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 6 FOLLOWER]: Advancing to term 7
I20251214 20:52:33.250140 1203 tablet_bootstrap.cc:492] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.250492 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "082ec70b8ee646b898d9201ad073d292" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:33.251150 848 leader_election.cc:343] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.251654 2937 leader_election.cc:290] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819)
I20251214 20:52:33.252156 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "082ec70b8ee646b898d9201ad073d292" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.252347 1050 raft_consensus.cc:3060] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.252658 1334 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6713a20a82b434f8a8bc122113e26ad in term 7.
I20251214 20:52:33.253408 2291 leader_election.cc:304] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4, e6713a20a82b434f8a8bc122113e26ad; no voters:
I20251214 20:52:33.253428 2937 raft_consensus.cc:2764] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20251214 20:52:33.253824 2936 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 7 FOLLOWER]: Leader election won for term 7
I20251214 20:52:33.253924 1203 tablet_bootstrap.cc:654] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.258553 1050 raft_consensus.cc:2468] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:33.259390 2889 leader_election.cc:290] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.260352 2777 leader_election.cc:290] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.259263 2735 raft_consensus.cc:493] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.271452 2735 raft_consensus.cc:515] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.272382 2735 leader_election.cc:290] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.273561 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "161fa219cc2f4c44bbcfbdfe9dce9a1f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.274121 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "161fa219cc2f4c44bbcfbdfe9dce9a1f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.274199 991 leader_election.cc:343] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.274863 2884 raft_consensus.cc:493] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:33.274919 989 leader_election.cc:343] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.275012 989 leader_election.cc:304] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.275043 2884 raft_consensus.cc:515] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.275453 2735 raft_consensus.cc:2749] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.275673 2777 raft_consensus.cc:493] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.275720 2884 leader_election.cc:290] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.275812 2777 raft_consensus.cc:515] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:33.277145 713 leader_election.cc:343] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 3d949abef1cb41a9905482d81a98d2c1
W20251214 20:52:33.277298 712 leader_election.cc:343] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 3d949abef1cb41a9905482d81a98d2c1
I20251214 20:52:33.277393 712 leader_election.cc:304] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.277853 2884 raft_consensus.cc:2749] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.278591 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a57409d4f54f17be792e6f3f17d7b7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.279094 848 leader_election.cc:343] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.279944 2777 leader_election.cc:290] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.280053 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a57409d4f54f17be792e6f3f17d7b7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.280309 1051 raft_consensus.cc:2468] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:33.280545 851 leader_election.cc:304] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.281149 2735 raft_consensus.cc:493] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.281286 2735 raft_consensus.cc:515] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.281999 2735 leader_election.cc:290] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.282876 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f04336d2c84439b9003df19d3541273" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.283767 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f04336d2c84439b9003df19d3541273" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.284466 991 leader_election.cc:343] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.284873 989 leader_election.cc:343] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.284963 989 leader_election.cc:304] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.285132 1202 tablet_bootstrap.cc:492] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.285271 1202 ts_tablet_manager.cc:1403] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.190s user 0.015s sys 0.031s
I20251214 20:52:33.285460 2735 raft_consensus.cc:2749] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.285784 851 leader_election.cc:304] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.286032 1202 raft_consensus.cc:359] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.286197 1202 raft_consensus.cc:385] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.286262 1202 raft_consensus.cc:740] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.286375 2889 raft_consensus.cc:2804] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.286516 2889 raft_consensus.cc:697] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.286777 2889 consensus_queue.cc:237] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.287871 2786 raft_consensus.cc:2804] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.287983 2786 raft_consensus.cc:493] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.288105 2786 raft_consensus.cc:3060] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.290535 2786 raft_consensus.cc:515] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.291087 2936 raft_consensus.cc:697] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 7 LEADER]: Becoming Leader. State: Replica: e6713a20a82b434f8a8bc122113e26ad, State: Running, Role: LEADER
I20251214 20:52:33.291888 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a57409d4f54f17be792e6f3f17d7b7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:33.291862 2936 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 945, Committed index: 945, Last appended: 4.955, Last appended by leader: 955, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:33.286476 1202 consensus_queue.cc:260] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:33.292500 848 leader_election.cc:343] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.293064 1202 ts_tablet_manager.cc:1434] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251214 20:52:33.293529 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2a57409d4f54f17be792e6f3f17d7b7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.293733 1051 raft_consensus.cc:3060] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.295686 1203 tablet_bootstrap.cc:492] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.295838 1203 ts_tablet_manager.cc:1403] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.015s
I20251214 20:52:33.296571 1203 raft_consensus.cc:359] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.296725 1203 raft_consensus.cc:385] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.296765 1203 raft_consensus.cc:740] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.296901 1051 raft_consensus.cc:2468] T d2a57409d4f54f17be792e6f3f17d7b7 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:33.296972 1203 consensus_queue.cc:260] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.299906 1203 ts_tablet_manager.cc:1434] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:33.300192 1203 tablet_bootstrap.cc:492] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.301975 1203 tablet_bootstrap.cc:654] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.303194 1206 tablet_bootstrap.cc:492] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.303442 1206 ts_tablet_manager.cc:1403] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.117s user 0.007s sys 0.014s
I20251214 20:52:33.304349 1206 raft_consensus.cc:359] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.304765 1206 raft_consensus.cc:385] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.304935 1206 raft_consensus.cc:740] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.305315 1206 consensus_queue.cc:260] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.309520 2941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6713a20a82b434f8a8bc122113e26ad. Latest consensus state: current_term: 7 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:33.310091 2941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:33.310798 1206 ts_tablet_manager.cc:1434] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:33.293370 1202 tablet_bootstrap.cc:492] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.293476 2786 leader_election.cc:290] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.297353 851 leader_election.cc:304] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.309825 2889 raft_consensus.cc:493] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.311949 2889 raft_consensus.cc:515] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.312644 2889 leader_election.cc:290] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.312767 2889 raft_consensus.cc:2804] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.312872 2889 raft_consensus.cc:697] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.313165 2889 consensus_queue.cc:237] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.314082 1202 tablet_bootstrap.cc:654] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.314913 2890 raft_consensus.cc:493] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.315037 2890 raft_consensus.cc:515] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.315644 2890 leader_election.cc:290] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.315958 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "140667afe42f42659a8cd38ed2517201" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.316819 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1d0e555eee4c56be271d14c34335bd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.317860 848 leader_election.cc:343] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.318075 848 leader_election.cc:343] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.318439 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1d0e555eee4c56be271d14c34335bd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.318657 1050 raft_consensus.cc:2468] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:33.319166 851 leader_election.cc:304] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.319649 2786 raft_consensus.cc:2804] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.319728 2786 raft_consensus.cc:493] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.319789 2786 raft_consensus.cc:3060] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.321956 1203 tablet_bootstrap.cc:492] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.322096 1203 ts_tablet_manager.cc:1403] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20251214 20:52:33.322036 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "140667afe42f42659a8cd38ed2517201" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.322243 1051 raft_consensus.cc:2468] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:33.322624 2786 raft_consensus.cc:515] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.322726 1206 tablet_bootstrap.cc:492] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.322755 1203 raft_consensus.cc:359] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.322909 1203 raft_consensus.cc:385] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.322948 1203 raft_consensus.cc:740] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.323189 1203 consensus_queue.cc:260] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.323704 851 leader_election.cc:304] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.323751 1203 ts_tablet_manager.cc:1434] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:33.324174 2777 raft_consensus.cc:2804] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.324252 2777 raft_consensus.cc:493] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.324318 2777 raft_consensus.cc:3060] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.290629 1977 ts_tablet_manager.cc:1929] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:33.324568 1977 log.cc:1199] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/06381efe75284e649fd6d1a9903cfa9b
I20251214 20:52:33.324885 1977 ts_tablet_manager.cc:1950] T 06381efe75284e649fd6d1a9903cfa9b P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:33.325698 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 547) took 1756 ms. Trace:
I20251214 20:52:33.325783 1977 rpcz_store.cc:276] 1214 20:52:31.568931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.569014 (+ 83us) service_pool.cc:225] Handling call
1214 20:52:33.050904 (+1481890us) ts_tablet_manager.cc:1151] Deleting tablet 06381efe75284e649fd6d1a9903cfa9b
1214 20:52:33.050912 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.287163 (+236251us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.290510 (+ 3347us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.325676 (+ 35166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1481749}
I20251214 20:52:33.326220 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 548) took 1757 ms. Trace:
I20251214 20:52:33.326295 1977 rpcz_store.cc:276] 1214 20:52:31.569006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:31.569184 (+ 178us) service_pool.cc:225] Handling call
1214 20:52:33.326111 (+1756927us) ts_tablet_manager.cc:1151] Deleting tablet 47e813373fbd482298312ead4f037c01
1214 20:52:33.326121 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.326203 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1756806}
I20251214 20:52:33.326586 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 562) took 1318 ms. Trace:
I20251214 20:52:33.326642 1977 rpcz_store.cc:276] 1214 20:52:32.008571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:32.008655 (+ 84us) service_pool.cc:225] Handling call
1214 20:52:33.326490 (+1317835us) ts_tablet_manager.cc:1151] Deleting tablet 5b68367f902f4be386788a8d138b5762
1214 20:52:33.326498 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.326573 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1317514}
I20251214 20:52:33.326903 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 563) took 1316 ms. Trace:
I20251214 20:52:33.326959 1977 rpcz_store.cc:276] 1214 20:52:32.010396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:32.010446 (+ 50us) service_pool.cc:225] Handling call
1214 20:52:33.326818 (+1316372us) ts_tablet_manager.cc:1151] Deleting tablet 0b7dd0ab78534a34ba342ec6e4fae129
1214 20:52:33.326825 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.326889 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1316216}
I20251214 20:52:33.327244 1977 tablet_replica.cc:333] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:33.327481 1977 raft_consensus.cc:2243] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:33.327632 1977 raft_consensus.cc:2272] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:33.327720 1206 tablet_bootstrap.cc:654] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.328234 1977 ts_tablet_manager.cc:1916] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:33.329393 2884 raft_consensus.cc:493] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.329551 2884 raft_consensus.cc:515] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.330209 2884 leader_election.cc:290] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.330857 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d157ac8f74eb7911df05ab5e4a3d5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.331117 1051 raft_consensus.cc:2393] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 074399864fb2470fa9e2ee87c1fe2212 in current term 1: Already voted for candidate a2e8fd1cb731446d88f8e510c7a52999 in this term.
I20251214 20:52:33.331210 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "719d157ac8f74eb7911df05ab5e4a3d5" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.334758 2777 raft_consensus.cc:515] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.335887 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "140667afe42f42659a8cd38ed2517201" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:33.336437 848 leader_election.cc:343] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.336761 2777 leader_election.cc:290] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.338433 1203 tablet_bootstrap.cc:492] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.339077 1977 ts_tablet_manager.cc:1929] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:33.339211 1977 log.cc:1199] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/f84e9dfc23d34eb0a379e8ac5a442e59
I20251214 20:52:33.339504 1977 ts_tablet_manager.cc:1950] T f84e9dfc23d34eb0a379e8ac5a442e59 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:33.340672 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1d0e555eee4c56be271d14c34335bd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:33.341254 1203 tablet_bootstrap.cc:654] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:33.341486 848 leader_election.cc:343] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.341708 2786 leader_election.cc:290] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.343096 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 564) took 1318 ms. Trace:
I20251214 20:52:33.343206 1977 rpcz_store.cc:276] 1214 20:52:32.024893 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:32.024963 (+ 70us) service_pool.cc:225] Handling call
1214 20:52:33.327193 (+1302230us) ts_tablet_manager.cc:1151] Deleting tablet f84e9dfc23d34eb0a379e8ac5a442e59
1214 20:52:33.327202 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.334180 (+ 6978us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.338955 (+ 4775us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.343073 (+ 4118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1302009}
I20251214 20:52:33.343835 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 566) took 1294 ms. Trace:
I20251214 20:52:33.343991 1977 rpcz_store.cc:276] 1214 20:52:32.048980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:32.049036 (+ 56us) service_pool.cc:225] Handling call
1214 20:52:33.343701 (+1294665us) ts_tablet_manager.cc:1151] Deleting tablet c07966670d294a0495ec54d48c7f09cd
1214 20:52:33.343713 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.343819 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1294408}
I20251214 20:52:33.344393 1977 tablet_replica.cc:333] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:33.344625 1977 raft_consensus.cc:2243] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:33.344789 1977 raft_consensus.cc:2272] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:33.345656 1977 ts_tablet_manager.cc:1916] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:33.351886 1203 tablet_bootstrap.cc:492] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.352191 1203 ts_tablet_manager.cc:1403] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20251214 20:52:33.353142 1203 raft_consensus.cc:359] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.353366 1203 raft_consensus.cc:385] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.353416 1203 raft_consensus.cc:740] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.353631 1203 consensus_queue.cc:260] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.354967 2884 raft_consensus.cc:493] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.355185 2884 raft_consensus.cc:515] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.355965 2884 leader_election.cc:290] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:33.357105 713 leader_election.cc:343] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: f84e9dfc23d34eb0a379e8ac5a442e59
I20251214 20:52:33.354838 1203 ts_tablet_manager.cc:1434] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:33.357878 1203 tablet_bootstrap.cc:492] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.359273 2890 raft_consensus.cc:493] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.359440 2890 raft_consensus.cc:515] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.360033 2890 leader_election.cc:290] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.360404 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d7c7b3f62764ff497f4d70b53eed7de" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.361209 848 leader_election.cc:343] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.366591 2735 raft_consensus.cc:493] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.366860 2735 raft_consensus.cc:515] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.368183 2735 leader_election.cc:290] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.368939 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "727e7b61ae3942098712720d344c5310" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.369534 989 leader_election.cc:343] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.370205 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "727e7b61ae3942098712720d344c5310" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.370754 991 leader_election.cc:343] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.370850 991 leader_election.cc:304] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.371378 2735 raft_consensus.cc:2749] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.382647 1203 tablet_bootstrap.cc:654] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.392145 1977 ts_tablet_manager.cc:1929] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:33.392315 1977 log.cc:1199] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/e9f6da5d29294f08a86c96c6821dda10
I20251214 20:52:33.393513 712 leader_election.cc:304] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.394110 2884 raft_consensus.cc:3060] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.395942 2819 raft_consensus.cc:493] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.396145 2819 raft_consensus.cc:515] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.396850 2819 leader_election.cc:290] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.397883 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a23561b9f4e48bbbd796e14fafec086" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.398910 2819 raft_consensus.cc:493] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.399075 2819 raft_consensus.cc:515] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.399674 2819 leader_election.cc:290] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.399827 2735 raft_consensus.cc:493] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.399952 2735 raft_consensus.cc:515] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.400522 2735 leader_election.cc:290] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.401396 2884 raft_consensus.cc:2749] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.401500 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80098b7c24824d129b8f25daf2115460" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.402056 991 leader_election.cc:343] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.402777 2890 consensus_queue.cc:1048] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:33.403627 713 leader_election.cc:343] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 37a33cc9bb784f72956ac9f5493a0c3d
I20251214 20:52:33.403992 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80098b7c24824d129b8f25daf2115460" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.404234 848 consensus_peers.cc:597] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:33.404618 989 leader_election.cc:343] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.404723 989 leader_election.cc:304] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.405395 2735 raft_consensus.cc:2749] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.407312 2735 raft_consensus.cc:493] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.407442 2735 raft_consensus.cc:515] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.408090 2735 leader_election.cc:290] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.408696 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eb1006b0ba440dcba2f9de04dca2702" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.408908 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eb1006b0ba440dcba2f9de04dca2702" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.409372 991 leader_election.cc:343] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.409518 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "140667afe42f42659a8cd38ed2517201" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.409684 1051 raft_consensus.cc:3060] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.412750 1051 raft_consensus.cc:2468] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
W20251214 20:52:33.413245 989 leader_election.cc:343] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.413707 989 leader_election.cc:304] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.414688 2735 raft_consensus.cc:2749] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.417604 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f1d0e555eee4c56be271d14c34335bd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.417896 1051 raft_consensus.cc:3060] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.421027 1051 raft_consensus.cc:2468] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:33.425638 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a23561b9f4e48bbbd796e14fafec086" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.426666 712 leader_election.cc:343] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: f84e9dfc23d34eb0a379e8ac5a442e59
I20251214 20:52:33.426846 712 leader_election.cc:304] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.427390 2884 raft_consensus.cc:2749] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.427960 712 leader_election.cc:304] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.428399 2884 raft_consensus.cc:2749] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:33.428851 712 leader_election.cc:343] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 37a33cc9bb784f72956ac9f5493a0c3d
I20251214 20:52:33.428995 712 leader_election.cc:304] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.429680 2884 raft_consensus.cc:2749] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.430284 851 leader_election.cc:304] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.430856 2786 raft_consensus.cc:2804] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.430976 2786 raft_consensus.cc:697] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.431242 2786 consensus_queue.cc:237] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.435285 851 leader_election.cc:304] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.438791 2950 raft_consensus.cc:2804] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.439014 2950 raft_consensus.cc:697] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.439436 2950 consensus_queue.cc:237] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:33.445179 851 leader_election.cc:343] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 0d7c7b3f62764ff497f4d70b53eed7de
I20251214 20:52:33.446233 1206 tablet_bootstrap.cc:492] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.446308 1203 tablet_bootstrap.cc:492] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.446463 1203 ts_tablet_manager.cc:1403] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.003s
I20251214 20:52:33.446750 1206 ts_tablet_manager.cc:1403] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.124s user 0.009s sys 0.041s
I20251214 20:52:33.447204 851 leader_election.cc:304] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.445183 1202 tablet_bootstrap.cc:492] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.447489 2889 raft_consensus.cc:2749] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.447633 1202 ts_tablet_manager.cc:1403] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.154s user 0.010s sys 0.042s
I20251214 20:52:33.448424 1202 raft_consensus.cc:359] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.448815 1203 raft_consensus.cc:359] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.449007 1203 raft_consensus.cc:385] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.449070 1203 raft_consensus.cc:740] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.448974 1206 raft_consensus.cc:359] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.449417 1202 raft_consensus.cc:385] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.449525 1202 raft_consensus.cc:740] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.449678 1206 raft_consensus.cc:385] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.450032 1202 consensus_queue.cc:260] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.450299 1206 raft_consensus.cc:740] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.450457 1203 consensus_queue.cc:260] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.451084 1202 ts_tablet_manager.cc:1434] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:33.450583 1206 consensus_queue.cc:260] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.451820 1206 ts_tablet_manager.cc:1434] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:33.452159 1202 tablet_bootstrap.cc:492] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.452157 1203 ts_tablet_manager.cc:1434] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251214 20:52:33.452519 1206 tablet_bootstrap.cc:492] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.453159 1203 tablet_bootstrap.cc:492] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.454856 1206 tablet_bootstrap.cc:654] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.449561 1977 ts_tablet_manager.cc:1950] T e9f6da5d29294f08a86c96c6821dda10 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:33.455818 1977 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37530 (request call id 565) took 1407 ms. Trace:
I20251214 20:52:33.457015 1977 rpcz_store.cc:276] 1214 20:52:32.048851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:32.050609 (+ 1758us) service_pool.cc:225] Handling call
1214 20:52:33.344337 (+1293728us) ts_tablet_manager.cc:1151] Deleting tablet e9f6da5d29294f08a86c96c6821dda10
1214 20:52:33.344345 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1214 20:52:33.373165 (+ 28820us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.391991 (+ 18826us) tablet_metadata.cc:641] Metadata flushed
1214 20:52:33.455797 (+ 63806us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293570}
I20251214 20:52:33.456959 1202 tablet_bootstrap.cc:654] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.458091 2950 raft_consensus.cc:493] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.458367 2950 raft_consensus.cc:515] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.459282 2950 leader_election.cc:290] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.460732 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0976ffcc649461fac74e20b5e402275" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.466030 851 leader_election.cc:343] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: a0976ffcc649461fac74e20b5e402275
W20251214 20:52:33.470597 848 leader_election.cc:343] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.470774 848 leader_election.cc:304] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.473114 1203 tablet_bootstrap.cc:654] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.476684 2890 raft_consensus.cc:2749] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.476830 1206 tablet_bootstrap.cc:492] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.476990 1206 ts_tablet_manager.cc:1403] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.002s
I20251214 20:52:33.477790 1206 raft_consensus.cc:359] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.478832 1206 raft_consensus.cc:385] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.478919 1206 raft_consensus.cc:740] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.479178 1206 consensus_queue.cc:260] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.483019 2890 consensus_queue.cc:1048] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:33.485626 1202 tablet_bootstrap.cc:492] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.486017 1202 ts_tablet_manager.cc:1403] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.002s
I20251214 20:52:33.486790 1202 raft_consensus.cc:359] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.487049 1202 raft_consensus.cc:385] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.487159 1202 raft_consensus.cc:740] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.487533 1202 consensus_queue.cc:260] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.489043 1206 ts_tablet_manager.cc:1434] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251214 20:52:33.492362 1206 tablet_bootstrap.cc:492] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.495886 768 raft_consensus.cc:3060] T 876d2a1f1d1c4f6882a5857b43568ee5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.497437 1203 tablet_bootstrap.cc:492] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.497597 1203 ts_tablet_manager.cc:1403] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.001s
I20251214 20:52:33.498356 1203 raft_consensus.cc:359] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.498533 1203 raft_consensus.cc:385] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.498587 1203 raft_consensus.cc:740] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.498817 1203 consensus_queue.cc:260] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.499583 1203 ts_tablet_manager.cc:1434] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:33.499871 1203 tablet_bootstrap.cc:492] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.500957 2950 raft_consensus.cc:493] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.501242 2950 raft_consensus.cc:515] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.501752 1203 tablet_bootstrap.cc:654] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.502512 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "088c72a6f5014b6fbc6ee060b6b7a590" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.502591 2950 leader_election.cc:290] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:33.503087 848 leader_election.cc:343] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.503279 851 leader_election.cc:343] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 088c72a6f5014b6fbc6ee060b6b7a590
I20251214 20:52:33.503443 851 leader_election.cc:304] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.503866 2950 raft_consensus.cc:2749] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.504199 1206 tablet_bootstrap.cc:654] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.505268 1202 ts_tablet_manager.cc:1434] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251214 20:52:33.517978 1202 tablet_bootstrap.cc:492] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.520190 1202 tablet_bootstrap.cc:654] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.530403 1202 tablet_bootstrap.cc:492] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.530594 1202 ts_tablet_manager.cc:1403] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20251214 20:52:33.530812 1206 tablet_bootstrap.cc:492] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.531028 1206 ts_tablet_manager.cc:1403] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.006s
I20251214 20:52:33.532127 1202 raft_consensus.cc:359] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.532310 1202 raft_consensus.cc:385] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.532368 1202 raft_consensus.cc:740] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.532548 1202 consensus_queue.cc:260] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.533000 1202 ts_tablet_manager.cc:1434] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:33.533257 1202 tablet_bootstrap.cc:492] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.534991 1202 tablet_bootstrap.cc:654] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.537891 1202 tablet_bootstrap.cc:492] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.538022 1202 ts_tablet_manager.cc:1403] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251214 20:52:33.538614 1202 raft_consensus.cc:359] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.538760 1202 raft_consensus.cc:385] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.538800 1202 raft_consensus.cc:740] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.538970 1202 consensus_queue.cc:260] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.539829 1203 tablet_bootstrap.cc:492] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.540064 1203 ts_tablet_manager.cc:1403] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.004s
I20251214 20:52:33.541392 1202 ts_tablet_manager.cc:1434] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251214 20:52:33.542158 1202 tablet_bootstrap.cc:492] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.542488 2890 raft_consensus.cc:493] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.542654 2890 raft_consensus.cc:515] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.543416 2890 leader_election.cc:290] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.543678 1202 tablet_bootstrap.cc:654] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:33.545174 851 leader_election.cc:343] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: fe24e4f4e74040249068505058c15622
I20251214 20:52:33.545249 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe24e4f4e74040249068505058c15622" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.547467 848 leader_election.cc:343] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.547596 848 leader_election.cc:304] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.548072 2889 raft_consensus.cc:2749] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.548454 2735 raft_consensus.cc:493] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.548606 2735 raft_consensus.cc:515] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.549268 2735 leader_election.cc:290] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.550206 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a07d94d5a431488489ee9fbcc113a9d7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.550856 989 leader_election.cc:343] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.551200 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a07d94d5a431488489ee9fbcc113a9d7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.551800 991 leader_election.cc:343] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.551896 991 leader_election.cc:304] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.552233 1203 raft_consensus.cc:359] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.552407 1203 raft_consensus.cc:385] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.552455 1203 raft_consensus.cc:740] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.552639 1203 consensus_queue.cc:260] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.552335 2735 raft_consensus.cc:2749] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.553184 1203 ts_tablet_manager.cc:1434] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.013s user 0.000s sys 0.000s
I20251214 20:52:33.555074 1202 tablet_bootstrap.cc:492] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.555254 1202 ts_tablet_manager.cc:1403] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20251214 20:52:33.555955 1202 raft_consensus.cc:359] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.556126 1202 raft_consensus.cc:385] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.556735 1202 raft_consensus.cc:740] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.557092 1203 tablet_bootstrap.cc:492] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.557713 1202 consensus_queue.cc:260] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.559238 1203 tablet_bootstrap.cc:654] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.562048 1202 ts_tablet_manager.cc:1434] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251214 20:52:33.562335 1202 tablet_bootstrap.cc:492] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.569604 1202 tablet_bootstrap.cc:654] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.577971 1202 tablet_bootstrap.cc:492] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.578123 1202 ts_tablet_manager.cc:1403] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20251214 20:52:33.578565 1206 raft_consensus.cc:359] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.578806 1206 raft_consensus.cc:385] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.578859 1206 raft_consensus.cc:740] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.578766 1202 raft_consensus.cc:359] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.578924 1202 raft_consensus.cc:385] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.578969 1202 raft_consensus.cc:740] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.579202 1202 consensus_queue.cc:260] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.582322 2906 raft_consensus.cc:493] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.582466 2906 raft_consensus.cc:515] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.582732 1202 ts_tablet_manager.cc:1434] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:33.583045 1202 tablet_bootstrap.cc:492] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.583079 2906 leader_election.cc:290] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.583839 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e59b80ee919b4e1393fe317ed0e3a3d2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.584651 1202 tablet_bootstrap.cc:654] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.585358 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e59b80ee919b4e1393fe317ed0e3a3d2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.588918 989 leader_election.cc:343] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.589270 991 leader_election.cc:343] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.589380 991 leader_election.cc:304] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.589783 2906 raft_consensus.cc:2749] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.590998 1203 tablet_bootstrap.cc:492] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.591284 1203 ts_tablet_manager.cc:1403] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.006s
I20251214 20:52:33.592087 776 raft_consensus.cc:3060] T e6bdfbe4d15345de8be5de00bfe4ca07 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.593124 1203 raft_consensus.cc:359] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.593292 1203 raft_consensus.cc:385] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.593349 1203 raft_consensus.cc:740] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.594259 1203 consensus_queue.cc:260] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.594923 768 raft_consensus.cc:3060] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.601581 2889 raft_consensus.cc:493] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.601768 2889 raft_consensus.cc:515] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.602481 2889 leader_election.cc:290] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.603500 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae948ffc81764692b92045f5befe0184" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.603963 1203 ts_tablet_manager.cc:1434] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
W20251214 20:52:33.604008 851 leader_election.cc:343] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: ae948ffc81764692b92045f5befe0184
W20251214 20:52:33.604238 848 leader_election.cc:343] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.604317 1203 tablet_bootstrap.cc:492] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.604347 848 leader_election.cc:304] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.604851 2889 raft_consensus.cc:2749] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.606421 1203 tablet_bootstrap.cc:654] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.608367 1206 consensus_queue.cc:260] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.610275 1206 ts_tablet_manager.cc:1434] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.033s user 0.002s sys 0.002s
I20251214 20:52:33.610626 1206 tablet_bootstrap.cc:492] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.612730 1206 tablet_bootstrap.cc:654] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.613680 1202 tablet_bootstrap.cc:492] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.613834 1202 ts_tablet_manager.cc:1403] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.002s
I20251214 20:52:33.614625 1202 raft_consensus.cc:359] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.614799 1202 raft_consensus.cc:385] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.614846 1202 raft_consensus.cc:740] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.615053 1202 consensus_queue.cc:260] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.624434 1202 ts_tablet_manager.cc:1434] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251214 20:52:33.624871 1202 tablet_bootstrap.cc:492] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:33.625718 851 consensus_peers.cc:597] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6ebfc373fbf41b1920c271a15302484. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:33.626966 1202 tablet_bootstrap.cc:654] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:33.630481 848 consensus_peers.cc:597] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:33.632697 1203 tablet_bootstrap.cc:492] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.632885 1203 ts_tablet_manager.cc:1403] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.004s
I20251214 20:52:33.633690 1203 raft_consensus.cc:359] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.633893 1203 raft_consensus.cc:385] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.633950 1203 raft_consensus.cc:740] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.634183 1203 consensus_queue.cc:260] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.634845 1203 ts_tablet_manager.cc:1434] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:33.635578 1202 tablet_bootstrap.cc:492] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.635725 1202 ts_tablet_manager.cc:1403] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20251214 20:52:33.636422 1202 raft_consensus.cc:359] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.636595 1202 raft_consensus.cc:385] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.636647 1202 raft_consensus.cc:740] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.636833 1202 consensus_queue.cc:260] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.639396 1203 tablet_bootstrap.cc:492] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.641173 1203 tablet_bootstrap.cc:654] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.642877 1206 tablet_bootstrap.cc:492] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.643112 1202 ts_tablet_manager.cc:1434] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251214 20:52:33.643260 1206 ts_tablet_manager.cc:1403] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.004s
I20251214 20:52:33.643507 1202 tablet_bootstrap.cc:492] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.644114 1206 raft_consensus.cc:359] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.644317 1206 raft_consensus.cc:385] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.644392 1206 raft_consensus.cc:740] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.644641 1206 consensus_queue.cc:260] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:33.646359 440 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6e005feeb95b48b9ad33612ab81d5bce (table default.Table_ec8c8f9185be4470a6580c4e141e2227 [id=8872bfac95c64cbba1ccb84d1d9f2483], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6e005feeb95b48b9ad33612ab81d5bce consensus state has no leader
W20251214 20:52:33.649479 485 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:50082 (request call id 33) took 970 ms (client timeout 1000 ms). Trace:
W20251214 20:52:33.649729 485 rpcz_store.cc:269] 1214 20:52:32.679074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:32.679397 (+ 323us) service_pool.cc:225] Handling call
1214 20:52:33.649399 (+970002us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 20:52:33.652894 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu with pid 424
I20251214 20:52:33.654605 2397 raft_consensus.cc:1240] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [term 7 LEADER]: Rejecting Update request from peer c27224391721467a8f46e467a980b37a for earlier term 6. Current term is 7. Ops: []
I20251214 20:52:33.655372 2906 raft_consensus.cc:493] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.655507 2906 raft_consensus.cc:515] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.656046 2906 leader_election.cc:290] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.656824 1202 tablet_bootstrap.cc:654] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.657200 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f9def8e4fd449599820a63845397750" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.657622 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f9def8e4fd449599820a63845397750" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.658176 1206 ts_tablet_manager.cc:1434] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
W20251214 20:52:33.658413 991 leader_election.cc:343] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.659444 989 leader_election.cc:343] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.659550 989 leader_election.cc:304] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.660095 2906 raft_consensus.cc:2749] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.660969 1206 tablet_bootstrap.cc:492] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.662649 1206 tablet_bootstrap.cc:654] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.663389 2889 raft_consensus.cc:493] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.663573 2889 raft_consensus.cc:515] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.664166 2889 leader_election.cc:290] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.664306 1203 tablet_bootstrap.cc:492] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.664480 1203 ts_tablet_manager.cc:1403] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20251214 20:52:33.664661 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b47ca1a03944cf1847d3578856dafa7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.665117 848 leader_election.cc:343] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.665323 1203 raft_consensus.cc:359] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.665515 1203 raft_consensus.cc:385] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.665565 1203 raft_consensus.cc:740] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.665797 1203 consensus_queue.cc:260] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.666334 1203 ts_tablet_manager.cc:1434] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:33.666661 1203 tablet_bootstrap.cc:492] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:33.667253 851 leader_election.cc:343] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 3b47ca1a03944cf1847d3578856dafa7
I20251214 20:52:33.667371 851 leader_election.cc:304] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.667733 2889 raft_consensus.cc:2749] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.668388 1203 tablet_bootstrap.cc:654] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.670574 1206 tablet_bootstrap.cc:492] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.670723 1206 ts_tablet_manager.cc:1403] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20251214 20:52:33.671563 1206 raft_consensus.cc:359] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.671738 1206 raft_consensus.cc:385] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.671787 1206 raft_consensus.cc:740] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.672317 1206 consensus_queue.cc:260] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:33.674032 851 consensus_peers.cc:597] T 65e79db74fa046698507c83416ee1669 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65e79db74fa046698507c83416ee1669. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:33.675235 1202 tablet_bootstrap.cc:492] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.675400 1202 ts_tablet_manager.cc:1403] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20251214 20:52:33.676157 1202 raft_consensus.cc:359] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.676337 1202 raft_consensus.cc:385] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.676398 1202 raft_consensus.cc:740] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.677382 1202 consensus_queue.cc:260] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.677875 1206 ts_tablet_manager.cc:1434] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:33.677897 1202 ts_tablet_manager.cc:1434] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:33.678216 1202 tablet_bootstrap.cc:492] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.678234 1206 tablet_bootstrap.cc:492] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.680148 1202 tablet_bootstrap.cc:654] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.681911 2906 raft_consensus.cc:493] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.682060 2906 raft_consensus.cc:515] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.682156 1206 tablet_bootstrap.cc:654] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.682677 2906 leader_election.cc:290] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.683794 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8deabfdbd0674b2884ee5f82fa5e332f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.684346 989 leader_election.cc:343] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.684825 1203 tablet_bootstrap.cc:492] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.684966 1203 ts_tablet_manager.cc:1403] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251214 20:52:33.685602 1203 raft_consensus.cc:359] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.685752 1203 raft_consensus.cc:385] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.685791 1203 raft_consensus.cc:740] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.685989 1203 consensus_queue.cc:260] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.686479 1203 ts_tablet_manager.cc:1434] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20251214 20:52:33.688959 848 consensus_peers.cc:597] T e6ebfc373fbf41b1920c271a15302484 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:33.690193 1203 tablet_bootstrap.cc:492] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.691653 2906 raft_consensus.cc:493] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.691804 1203 tablet_bootstrap.cc:654] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.691773 2906 raft_consensus.cc:515] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.692366 2906 leader_election.cc:290] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.692929 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3915abc53b084854912a8e4119eefc7e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.693089 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3915abc53b084854912a8e4119eefc7e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.693557 991 leader_election.cc:343] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.693750 989 leader_election.cc:343] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.693831 989 leader_election.cc:304] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.694288 2906 raft_consensus.cc:2749] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.683475 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8deabfdbd0674b2884ee5f82fa5e332f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.698020 908 raft_consensus.cc:2468] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 0.
I20251214 20:52:33.699008 991 leader_election.cc:304] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.699934 2735 raft_consensus.cc:2804] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.700016 2735 raft_consensus.cc:493] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.700073 2735 raft_consensus.cc:3060] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.700443 1206 tablet_bootstrap.cc:492] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.700583 1206 ts_tablet_manager.cc:1403] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20251214 20:52:33.701340 1206 raft_consensus.cc:359] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.701514 1206 raft_consensus.cc:385] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.701557 1206 raft_consensus.cc:740] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.701751 1206 consensus_queue.cc:260] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.702903 2735 raft_consensus.cc:515] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.703440 2906 raft_consensus.cc:493] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.703553 2906 raft_consensus.cc:515] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.704098 2906 leader_election.cc:290] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.704269 1206 ts_tablet_manager.cc:1434] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251214 20:52:33.704564 1206 tablet_bootstrap.cc:492] T d424e3387698434f92e84eb1d43f5949 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.704923 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "566c6e4761d5486d964e276d151e32b0" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.705444 2735 leader_election.cc:290] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.705677 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "566c6e4761d5486d964e276d151e32b0" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.705979 991 leader_election.cc:343] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.706461 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8deabfdbd0674b2884ee5f82fa5e332f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:33.706621 908 raft_consensus.cc:3060] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.706658 1206 tablet_bootstrap.cc:654] T d424e3387698434f92e84eb1d43f5949 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.708328 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8deabfdbd0674b2884ee5f82fa5e332f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:33.717756 989 leader_election.cc:343] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.717971 989 leader_election.cc:304] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:33.718518 989 leader_election.cc:343] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.723412 2735 raft_consensus.cc:2749] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.723943 908 raft_consensus.cc:2468] T 8deabfdbd0674b2884ee5f82fa5e332f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63848a9fb78a4013a6f1657621272d75 in term 1.
I20251214 20:52:33.724522 991 leader_election.cc:304] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.725044 2735 raft_consensus.cc:2804] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.725219 2884 raft_consensus.cc:493] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.725447 2884 raft_consensus.cc:515] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.726166 2884 leader_election.cc:290] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.726281 1202 tablet_bootstrap.cc:492] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.726424 1202 ts_tablet_manager.cc:1403] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.008s
I20251214 20:52:33.727201 1202 raft_consensus.cc:359] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.727380 1202 raft_consensus.cc:385] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.727430 1202 raft_consensus.cc:740] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.727650 1202 consensus_queue.cc:260] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:33.727813 713 leader_election.cc:343] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 9e1835374cfa48939d72476870accc3b
I20251214 20:52:33.728289 1202 ts_tablet_manager.cc:1434] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:33.729816 2906 raft_consensus.cc:493] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.730052 2906 raft_consensus.cc:515] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.730736 2906 leader_election.cc:290] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.731288 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e11723c042a4637839f1b13bed0935a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.731544 1202 tablet_bootstrap.cc:492] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.731738 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1835374cfa48939d72476870accc3b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.731989 908 raft_consensus.cc:2468] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:33.732483 712 leader_election.cc:304] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.732959 2884 raft_consensus.cc:2804] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.733033 2884 raft_consensus.cc:493] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.733091 2884 raft_consensus.cc:3060] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.734198 1202 tablet_bootstrap.cc:654] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:33.735292 991 leader_election.cc:343] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.742465 1202 tablet_bootstrap.cc:492] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.742656 1202 ts_tablet_manager.cc:1403] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20251214 20:52:33.742894 2884 raft_consensus.cc:515] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.743767 1202 raft_consensus.cc:359] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.743935 1202 raft_consensus.cc:385] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.743989 1202 raft_consensus.cc:740] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.744330 1203 tablet_bootstrap.cc:492] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.744484 1203 ts_tablet_manager.cc:1403] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.054s user 0.030s sys 0.018s
I20251214 20:52:33.744597 2884 leader_election.cc:290] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.744247 1202 consensus_queue.cc:260] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.745123 1203 raft_consensus.cc:359] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.745288 1203 raft_consensus.cc:385] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.745338 1203 raft_consensus.cc:740] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:33.745429 713 leader_election.cc:343] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 9e1835374cfa48939d72476870accc3b
I20251214 20:52:33.745519 1202 ts_tablet_manager.cc:1434] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:33.745498 1203 consensus_queue.cc:260] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.745796 1202 tablet_bootstrap.cc:492] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.745954 1203 ts_tablet_manager.cc:1434] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 20:52:33.746218 1203 tablet_bootstrap.cc:492] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.747778 1203 tablet_bootstrap.cc:654] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.747822 1202 tablet_bootstrap.cc:654] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.748639 1206 tablet_bootstrap.cc:492] T d424e3387698434f92e84eb1d43f5949 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.748757 1206 ts_tablet_manager.cc:1403] T d424e3387698434f92e84eb1d43f5949 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.005s
I20251214 20:52:33.749365 1206 raft_consensus.cc:359] T d424e3387698434f92e84eb1d43f5949 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.749508 1206 raft_consensus.cc:385] T d424e3387698434f92e84eb1d43f5949 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.749550 1206 raft_consensus.cc:740] T d424e3387698434f92e84eb1d43f5949 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.749764 1206 consensus_queue.cc:260] T d424e3387698434f92e84eb1d43f5949 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.750200 1206 ts_tablet_manager.cc:1434] T d424e3387698434f92e84eb1d43f5949 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251214 20:52:33.750455 1206 tablet_bootstrap.cc:492] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.751950 1206 tablet_bootstrap.cc:654] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.756628 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e1835374cfa48939d72476870accc3b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:33.756953 2889 raft_consensus.cc:493] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.757165 2889 raft_consensus.cc:515] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.757841 2889 leader_election.cc:290] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.758332 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6eca34627bbc4a3f9915be72f3128370" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.758491 2889 raft_consensus.cc:493] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.758610 2889 raft_consensus.cc:515] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.758821 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6eca34627bbc4a3f9915be72f3128370" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:33.758893 848 leader_election.cc:343] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.759020 1051 raft_consensus.cc:2468] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:33.759119 2889 leader_election.cc:290] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.759667 851 leader_election.cc:304] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.759742 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e33ba1149a44c294885d76f676160a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.760138 848 leader_election.cc:343] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.760255 2889 raft_consensus.cc:2804] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.760314 2889 raft_consensus.cc:493] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.760285 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e33ba1149a44c294885d76f676160a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.760394 2889 raft_consensus.cc:3060] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.760466 1051 raft_consensus.cc:2468] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:33.762452 2890 raft_consensus.cc:493] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.762566 2890 raft_consensus.cc:515] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.763078 2890 leader_election.cc:290] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.763617 908 raft_consensus.cc:3060] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.763617 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f04336d2c84439b9003df19d3541273" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.764052 851 leader_election.cc:304] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.764287 2890 raft_consensus.cc:2804] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.764351 2890 raft_consensus.cc:493] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.764420 2890 raft_consensus.cc:3060] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.764732 2889 raft_consensus.cc:515] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.764734 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f04336d2c84439b9003df19d3541273" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.764945 1051 raft_consensus.cc:2468] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:33.765421 2889 leader_election.cc:290] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.765709 851 leader_election.cc:304] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:33.766199 1206 tablet_bootstrap.cc:492] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.766351 1206 ts_tablet_manager.cc:1403] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20251214 20:52:33.766348 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6eca34627bbc4a3f9915be72f3128370" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.766490 1051 raft_consensus.cc:3060] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.766999 1206 raft_consensus.cc:359] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.767164 1206 raft_consensus.cc:385] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.767071 2890 raft_consensus.cc:515] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.767210 1206 raft_consensus.cc:740] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.767411 1206 consensus_queue.cc:260] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.767700 2819 raft_consensus.cc:493] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.767836 2819 raft_consensus.cc:515] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.768451 2819 leader_election.cc:290] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.768662 2819 raft_consensus.cc:493] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.768750 2819 raft_consensus.cc:515] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.769173 2819 leader_election.cc:290] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.769237 1051 raft_consensus.cc:2468] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:33.770218 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6a8863ace4541aea7463af24db27142" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:33.770752 713 leader_election.cc:343] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 06381efe75284e649fd6d1a9903cfa9b
I20251214 20:52:33.771279 2889 raft_consensus.cc:2804] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.773218 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6eca34627bbc4a3f9915be72f3128370" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:33.773247 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e33ba1149a44c294885d76f676160a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:33.773928 1206 ts_tablet_manager.cc:1434] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20251214 20:52:33.774071 2889 raft_consensus.cc:493] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.774166 2889 raft_consensus.cc:3060] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:33.772467 848 leader_election.cc:343] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.774529 848 leader_election.cc:343] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.774685 848 leader_election.cc:343] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.775388 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e33ba1149a44c294885d76f676160a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.775573 1050 raft_consensus.cc:3060] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:33.777809 849 connection.cc:537] server connection from 127.0.0.1:37530 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251214 20:52:33.777855 2288 connection.cc:537] server connection from 127.0.0.1:41536 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251214 20:52:33.778283 1050 raft_consensus.cc:2468] T c8e33ba1149a44c294885d76f676160a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
W20251214 20:52:33.778654 848 connection.cc:537] client connection to 127.0.104.62:42865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251214 20:52:33.778622 1287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.62:42865 (error 108)
W20251214 20:52:33.779470 710 connection.cc:537] client connection to 127.0.104.62:42865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251214 20:52:33.779680 1206 tablet_bootstrap.cc:492] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.780133 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c6a8863ace4541aea7463af24db27142" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.780490 851 leader_election.cc:304] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.780928 2950 raft_consensus.cc:2804] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.781062 2950 raft_consensus.cc:697] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.781239 1206 tablet_bootstrap.cc:654] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.781333 2950 consensus_queue.cc:237] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.782655 851 leader_election.cc:304] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.782866 2786 raft_consensus.cc:2804] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.782958 2786 raft_consensus.cc:697] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.783177 2786 consensus_queue.cc:237] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:33.785161 712 leader_election.cc:343] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 06381efe75284e649fd6d1a9903cfa9b
I20251214 20:52:33.785305 712 leader_election.cc:304] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:33.785463 713 connection.cc:537] server connection from 127.0.0.1:57406 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251214 20:52:33.785706 712 leader_election.cc:304] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:33.785974 2288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.62:42865 (error 108)
I20251214 20:52:33.786214 2819 raft_consensus.cc:2749] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.786370 2819 raft_consensus.cc:2749] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.772706 2890 leader_election.cc:290] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.799768 2889 raft_consensus.cc:515] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.801209 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f04336d2c84439b9003df19d3541273" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.801483 1050 raft_consensus.cc:3060] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.803203 2889 leader_election.cc:290] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.803470 908 raft_consensus.cc:2468] T 9e1835374cfa48939d72476870accc3b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:33.803936 712 leader_election.cc:304] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.804633 1050 raft_consensus.cc:2468] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:33.805300 2884 raft_consensus.cc:2804] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.805441 2884 raft_consensus.cc:697] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:33.805703 2906 raft_consensus.cc:493] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.805724 2884 consensus_queue.cc:237] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:33.805918 2906 raft_consensus.cc:515] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.806608 2906 leader_election.cc:290] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.807615 851 leader_election.cc:304] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:33.808368 2889 raft_consensus.cc:2804] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.808523 2889 raft_consensus.cc:697] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.808794 2889 consensus_queue.cc:237] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.811494 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "054cfd587d19409fbdf7f4848bcaf3f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.813135 991 leader_election.cc:343] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.815239 2998 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 20:52:33.786929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:42865 (local address 127.0.104.2:36883)
1214 20:52:33.814847 (+ 27918us) negotiation.cc:107] Waiting for socket to connect
1214 20:52:33.815001 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42865: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":27778,"thread_start_us":27656,"threads_started":1}
I20251214 20:52:33.815488 2890 raft_consensus.cc:493] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.815644 2890 raft_consensus.cc:515] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.816303 2890 leader_election.cc:290] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:33.817652 851 leader_election.cc:343] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: d01db63fd5d2474a9defdce0b98fc29d
I20251214 20:52:33.818315 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e11723c042a4637839f1b13bed0935a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.819190 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "054cfd587d19409fbdf7f4848bcaf3f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.822993 989 connection.cc:537] client connection to 127.0.104.62:42865 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251214 20:52:33.825131 989 leader_election.cc:343] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.825371 989 leader_election.cc:304] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.825661 2906 raft_consensus.cc:2749] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:33.826100 989 leader_election.cc:343] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.826274 989 leader_election.cc:304] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.826536 2906 raft_consensus.cc:2749] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.827787 2890 consensus_queue.cc:1048] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:33.829564 851 consensus_peers.cc:597] T b69bb4d5979f492683fc13238a668b9f P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b69bb4d5979f492683fc13238a668b9f. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:33.830242 2993 negotiation.cc:337] Failed RPC negotiation. Trace:
1214 20:52:33.780889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:42865 (local address 127.0.104.1:52457)
1214 20:52:33.829838 (+ 48949us) negotiation.cc:107] Waiting for socket to connect
1214 20:52:33.829974 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42865: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":48656,"thread_start_us":49130,"threads_started":1}
I20251214 20:52:33.834478 1203 tablet_bootstrap.cc:492] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:33.834654 1203 ts_tablet_manager.cc:1403] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.089s user 0.000s sys 0.020s
I20251214 20:52:33.835834 2735 raft_consensus.cc:697] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [term 1 LEADER]: Becoming Leader. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Running, Role: LEADER
I20251214 20:52:33.836402 1206 tablet_bootstrap.cc:492] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.836278 2735 consensus_queue.cc:237] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.836565 1206 ts_tablet_manager.cc:1403] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.010s
I20251214 20:52:33.837340 1206 raft_consensus.cc:359] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.837528 1206 raft_consensus.cc:385] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.837585 1206 raft_consensus.cc:740] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.837777 1206 consensus_queue.cc:260] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.838294 1206 ts_tablet_manager.cc:1434] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:33.838588 1206 tablet_bootstrap.cc:492] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.841140 1206 tablet_bootstrap.cc:654] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.844805 1203 raft_consensus.cc:359] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.845026 1203 raft_consensus.cc:385] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.845086 1203 raft_consensus.cc:740] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.845347 1203 consensus_queue.cc:260] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.845885 1203 ts_tablet_manager.cc:1434] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251214 20:52:33.846207 1203 tablet_bootstrap.cc:492] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:33.846926 1287 leader_election.cc:336] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c27224391721467a8f46e467a980b37a (127.0.104.62:42865): Network error: Client connection negotiation failed: client connection to 127.0.104.62:42865: connect: Connection refused (error 111)
I20251214 20:52:33.848021 1203 tablet_bootstrap.cc:654] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:33.850660 2288 leader_election.cc:336] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c27224391721467a8f46e467a980b37a (127.0.104.62:42865): Network error: Client connection negotiation failed: client connection to 127.0.104.62:42865: connect: Connection refused (error 111)
I20251214 20:52:33.851040 1203 tablet_bootstrap.cc:492] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
W20251214 20:52:33.851018 2288 leader_election.cc:336] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer c27224391721467a8f46e467a980b37a (127.0.104.62:42865): Network error: Client connection negotiation failed: client connection to 127.0.104.62:42865: connect: Connection refused (error 111)
I20251214 20:52:33.852265 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9f04336d2c84439b9003df19d3541273" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:33.852509 1203 ts_tablet_manager.cc:1403] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20251214 20:52:33.852828 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d01db63fd5d2474a9defdce0b98fc29d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.853173 1203 raft_consensus.cc:359] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.853331 1203 raft_consensus.cc:385] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.853391 1203 raft_consensus.cc:740] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:33.853883 848 consensus_peers.cc:597] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:33.854176 848 leader_election.cc:343] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.854698 848 leader_election.cc:343] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.854878 848 leader_election.cc:304] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212, 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.856076 2786 raft_consensus.cc:2749] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.856523 1203 consensus_queue.cc:260] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.857069 1206 tablet_bootstrap.cc:492] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.857198 1206 ts_tablet_manager.cc:1403] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20251214 20:52:33.857823 1206 raft_consensus.cc:359] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.857959 1206 raft_consensus.cc:385] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.858004 1206 raft_consensus.cc:740] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.858157 1206 consensus_queue.cc:260] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.858660 1206 ts_tablet_manager.cc:1434] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 20:52:33.859263 1203 ts_tablet_manager.cc:1434] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251214 20:52:33.859549 1203 tablet_bootstrap.cc:492] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:33.861138 1203 tablet_bootstrap.cc:654] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:33.863565 2288 consensus_peers.cc:597] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad -> Peer c27224391721467a8f46e467a980b37a (127.0.104.62:42865): Couldn't send request to peer c27224391721467a8f46e467a980b37a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:33.864740 1206 tablet_bootstrap.cc:492] T 4f1d0e555eee4c56be271d14c34335bd P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.866078 1206 tablet_bootstrap.cc:654] T 4f1d0e555eee4c56be271d14c34335bd P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.868680 1202 tablet_bootstrap.cc:492] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.868901 1202 ts_tablet_manager.cc:1403] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.123s user 0.006s sys 0.000s
I20251214 20:52:33.869719 1202 raft_consensus.cc:359] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.869916 1202 raft_consensus.cc:385] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.870004 1202 raft_consensus.cc:740] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.870014 2906 raft_consensus.cc:493] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.870162 2906 raft_consensus.cc:515] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.870334 1202 consensus_queue.cc:260] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.870796 2906 leader_election.cc:290] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.870855 1202 ts_tablet_manager.cc:1434] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:33.872587 1202 tablet_bootstrap.cc:492] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.872853 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bed2c326906486e99300e05033c9c0f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.875742 1206 tablet_bootstrap.cc:492] T 4f1d0e555eee4c56be271d14c34335bd P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:33.875886 1206 ts_tablet_manager.cc:1403] T 4f1d0e555eee4c56be271d14c34335bd P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
W20251214 20:52:33.876467 991 leader_election.cc:343] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.876519 1206 raft_consensus.cc:359] T 4f1d0e555eee4c56be271d14c34335bd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.876690 1206 raft_consensus.cc:385] T 4f1d0e555eee4c56be271d14c34335bd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.876746 1206 raft_consensus.cc:740] T 4f1d0e555eee4c56be271d14c34335bd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.876972 1206 consensus_queue.cc:260] T 4f1d0e555eee4c56be271d14c34335bd P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.877452 1206 ts_tablet_manager.cc:1434] T 4f1d0e555eee4c56be271d14c34335bd P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:33.880056 1206 tablet_bootstrap.cc:492] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:33.881139 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bed2c326906486e99300e05033c9c0f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.881716 1206 tablet_bootstrap.cc:654] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:33.882515 989 leader_election.cc:343] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.874426 1202 tablet_bootstrap.cc:654] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.891108 2786 raft_consensus.cc:493] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.891321 2786 raft_consensus.cc:515] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.891919 2786 leader_election.cc:290] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.896162 2786 raft_consensus.cc:493] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.896317 2786 raft_consensus.cc:515] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.896925 2786 leader_election.cc:290] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.897619 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80098b7c24824d129b8f25daf2115460" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.898186 848 leader_election.cc:343] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.898579 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e621f29c01448399d22615284fccd4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.898988 848 leader_election.cc:343] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.902093 2991 raft_consensus.cc:493] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.902249 2991 raft_consensus.cc:515] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.902863 2991 leader_election.cc:290] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:33.904634 713 leader_election.cc:343] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 274185a4115b458e8e9a376bdbcc870b
I20251214 20:52:33.908469 1334 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 7 FOLLOWER]: Refusing update from remote peer e6713a20a82b434f8a8bc122113e26ad: Log matching property violated. Preceding OpId in replica: term: 4 index: 955. Preceding OpId from leader: term: 7 index: 956. (index mismatch)
I20251214 20:52:33.908583 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274185a4115b458e8e9a376bdbcc870b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.908819 908 raft_consensus.cc:2468] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:33.909320 712 leader_election.cc:304] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.909386 3006 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 956, Last known committed idx: 955, Time since last communication: 0.000s
I20251214 20:52:33.909780 2991 raft_consensus.cc:2804] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.909852 2991 raft_consensus.cc:493] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.909909 2991 raft_consensus.cc:3060] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.912729 2991 raft_consensus.cc:515] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:33.914449 713 leader_election.cc:343] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 274185a4115b458e8e9a376bdbcc870b
I20251214 20:52:33.914613 2991 leader_election.cc:290] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.914862 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "274185a4115b458e8e9a376bdbcc870b" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:33.915086 908 raft_consensus.cc:3060] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.918103 908 raft_consensus.cc:2468] T 274185a4115b458e8e9a376bdbcc870b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:33.892545 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e621f29c01448399d22615284fccd4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.918666 712 leader_election.cc:304] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:33.918785 1051 raft_consensus.cc:2468] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:33.897886 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80098b7c24824d129b8f25daf2115460" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:33.919207 2991 raft_consensus.cc:2804] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.919191 1050 raft_consensus.cc:2468] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:33.919682 2991 raft_consensus.cc:697] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:33.919983 2991 consensus_queue.cc:237] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.921828 851 leader_election.cc:304] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.922282 851 leader_election.cc:304] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.922472 1202 tablet_bootstrap.cc:492] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.922564 2950 raft_consensus.cc:2804] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.922647 2786 raft_consensus.cc:2804] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:33.922708 2950 raft_consensus.cc:493] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.922791 2950 raft_consensus.cc:3060] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.923048 989 leader_election.cc:304] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.925429 2906 raft_consensus.cc:2749] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.925585 2950 raft_consensus.cc:515] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.922719 2786 raft_consensus.cc:493] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:33.926638 2786 raft_consensus.cc:3060] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.927333 2906 raft_consensus.cc:493] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.927472 2906 raft_consensus.cc:515] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.928069 2906 leader_election.cc:290] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.928727 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e4dc66681ef48479ae3cb95b091564a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:33.929291 2786 raft_consensus.cc:515] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:33.929349 991 leader_election.cc:343] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.930100 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e4dc66681ef48479ae3cb95b091564a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.930564 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e621f29c01448399d22615284fccd4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:33.930590 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80098b7c24824d129b8f25daf2115460" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:33.931051 2786 leader_election.cc:290] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:33.931092 848 leader_election.cc:343] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.931336 848 leader_election.cc:343] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.931931 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80098b7c24824d129b8f25daf2115460" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.932129 1050 raft_consensus.cc:3060] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.932226 2950 leader_election.cc:290] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:33.922616 1202 ts_tablet_manager.cc:1403] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.007s
I20251214 20:52:33.933394 1202 raft_consensus.cc:359] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.933549 1202 raft_consensus.cc:385] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.933596 1202 raft_consensus.cc:740] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.933787 1202 consensus_queue.cc:260] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.934449 1202 ts_tablet_manager.cc:1434] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:33.934733 1202 tablet_bootstrap.cc:492] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.935308 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e621f29c01448399d22615284fccd4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:33.935463 1051 raft_consensus.cc:3060] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:33.936424 1202 tablet_bootstrap.cc:654] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.937857 1051 raft_consensus.cc:2468] T 79e621f29c01448399d22615284fccd4 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:33.939182 851 leader_election.cc:304] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.939258 3015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6713a20a82b434f8a8bc122113e26ad. Latest consensus state: current_term: 7 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:33.939567 3015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:33.940006 2950 raft_consensus.cc:2804] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.940152 2950 raft_consensus.cc:697] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.940449 2950 consensus_queue.cc:237] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.941670 2906 raft_consensus.cc:493] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.941794 2906 raft_consensus.cc:515] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.942394 2906 leader_election.cc:290] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.942518 2906 raft_consensus.cc:493] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.942610 2906 raft_consensus.cc:515] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.943065 2906 leader_election.cc:290] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.944054 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fd5566a34e2a82009cc9ce6bdd37" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.944833 991 leader_election.cc:343] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.935075 1050 raft_consensus.cc:2468] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:33.945712 851 leader_election.cc:304] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:33.946107 2786 raft_consensus.cc:2804] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:33.946214 2786 raft_consensus.cc:697] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:33.946448 2786 consensus_queue.cc:237] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.947767 1202 tablet_bootstrap.cc:492] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:33.947942 1202 ts_tablet_manager.cc:1403] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20251214 20:52:33.948625 1202 raft_consensus.cc:359] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.948793 1202 raft_consensus.cc:385] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:33.948846 1202 raft_consensus.cc:740] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:33.957103 3015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:33.957434 3015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [sys.catalog]: This master's current role is: FOLLOWER
I20251214 20:52:33.960716 1202 consensus_queue.cc:260] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:33.961017 2985 raft_consensus.cc:493] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.961229 2985 raft_consensus.cc:515] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.961540 1202 ts_tablet_manager.cc:1434] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251214 20:52:33.961943 2985 leader_election.cc:290] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.962357 1202 tablet_bootstrap.cc:492] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:33.962627 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0ad1aa4193467db74cd707e7084b71" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.963308 991 leader_election.cc:343] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.963837 1202 tablet_bootstrap.cc:654] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:33.970387 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2688515bd1441fbad7516f8343ca775" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.970921 989 leader_election.cc:343] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:33.971352 848 consensus_peers.cc:597] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:33.971716 989 leader_election.cc:304] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:33.972245 991 leader_election.cc:343] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.973464 2735 raft_consensus.cc:2749] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.973495 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2688515bd1441fbad7516f8343ca775" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.974615 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fd5566a34e2a82009cc9ce6bdd37" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.976926 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0ad1aa4193467db74cd707e7084b71" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.978543 2735 raft_consensus.cc:493] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.978717 2735 raft_consensus.cc:515] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.979844 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5157ef5f1fd541efbbd2887bf4505254" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.980396 991 leader_election.cc:343] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.980973 2906 raft_consensus.cc:493] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:33.981117 2906 raft_consensus.cc:515] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:33.981711 2906 leader_election.cc:290] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:33.982316 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8db4364b5023438bb43538855007c95e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:33.982821 991 leader_election.cc:343] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.983070 2735 leader_election.cc:290] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:33.983315 989 leader_election.cc:343] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.984714 989 leader_election.cc:304] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:33.985208 989 leader_election.cc:343] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.985455 989 leader_election.cc:304] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.985380 2735 raft_consensus.cc:2749] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.985785 2906 raft_consensus.cc:2749] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:33.986159 989 leader_election.cc:343] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.986299 989 leader_election.cc:304] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.993332 2906 raft_consensus.cc:2749] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:33.994599 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5157ef5f1fd541efbbd2887bf4505254" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:33.995572 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8db4364b5023438bb43538855007c95e" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:33.997886 989 leader_election.cc:343] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:33.999258 989 leader_election.cc:304] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:33.999554 2906 raft_consensus.cc:2749] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:33.999991 989 leader_election.cc:343] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.000169 989 leader_election.cc:304] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.000406 2906 raft_consensus.cc:2749] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.000691 1203 tablet_bootstrap.cc:492] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.000861 1203 ts_tablet_manager.cc:1403] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.141s user 0.017s sys 0.021s
I20251214 20:52:34.001552 1203 raft_consensus.cc:359] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.001721 1203 raft_consensus.cc:385] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.001771 1203 raft_consensus.cc:740] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.001993 1203 consensus_queue.cc:260] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.010491 1206 tablet_bootstrap.cc:492] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.010470 1202 tablet_bootstrap.cc:492] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.010640 1202 ts_tablet_manager.cc:1403] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.003s
I20251214 20:52:34.010640 1206 ts_tablet_manager.cc:1403] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.131s user 0.005s sys 0.024s
I20251214 20:52:34.010735 1203 ts_tablet_manager.cc:1434] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251214 20:52:34.011341 1202 raft_consensus.cc:359] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.011545 1202 raft_consensus.cc:385] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.011606 1202 raft_consensus.cc:740] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.011808 1202 consensus_queue.cc:260] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.012292 1202 ts_tablet_manager.cc:1434] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:34.012603 1202 tablet_bootstrap.cc:492] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.013906 1202 tablet_bootstrap.cc:654] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.015870 1206 raft_consensus.cc:359] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.016045 1206 raft_consensus.cc:385] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.016099 1206 raft_consensus.cc:740] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.016304 1206 consensus_queue.cc:260] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.016768 1206 ts_tablet_manager.cc:1434] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.000s sys 0.000s
I20251214 20:52:34.017752 1206 tablet_bootstrap.cc:492] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.019899 1206 tablet_bootstrap.cc:654] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.030007 1202 tablet_bootstrap.cc:492] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.030166 1202 ts_tablet_manager.cc:1403] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.003s
I20251214 20:52:34.030974 1202 raft_consensus.cc:359] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.031160 1202 raft_consensus.cc:385] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.031224 1202 raft_consensus.cc:740] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.031446 1202 consensus_queue.cc:260] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.032495 1202 ts_tablet_manager.cc:1434] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:34.032770 1202 tablet_bootstrap.cc:492] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.034197 1202 tablet_bootstrap.cc:654] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.035737 1206 tablet_bootstrap.cc:492] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.035946 2906 raft_consensus.cc:493] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.036079 2906 raft_consensus.cc:515] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.036652 2906 leader_election.cc:290] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.037734 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825baba561ee4a8ebd44f75d8c51706a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.037976 1203 tablet_bootstrap.cc:492] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:34.038394 991 leader_election.cc:343] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.039928 1203 tablet_bootstrap.cc:654] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.039950 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825baba561ee4a8ebd44f75d8c51706a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:34.040563 989 leader_election.cc:343] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.040681 989 leader_election.cc:304] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.041306 2906 raft_consensus.cc:2749] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.042116 2889 consensus_queue.cc:1048] T f945814952b14131ba4e0c67a403a680 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.045414 1206 ts_tablet_manager.cc:1403] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20251214 20:52:34.046921 1206 raft_consensus.cc:359] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.048220 1206 raft_consensus.cc:385] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.048283 1202 tablet_bootstrap.cc:492] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.048424 1202 ts_tablet_manager.cc:1403] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20251214 20:52:34.048522 1206 raft_consensus.cc:740] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.048864 1206 consensus_queue.cc:260] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.049073 1202 raft_consensus.cc:359] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.049233 1202 raft_consensus.cc:385] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.049281 1202 raft_consensus.cc:740] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.049497 1202 consensus_queue.cc:260] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.051005 1206 ts_tablet_manager.cc:1434] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:34.052505 1206 tablet_bootstrap.cc:492] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.053128 1202 ts_tablet_manager.cc:1434] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:34.053989 1202 tablet_bootstrap.cc:492] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.055290 1202 tablet_bootstrap.cc:654] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.055297 1206 tablet_bootstrap.cc:654] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.060927 2991 raft_consensus.cc:493] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.061112 2991 raft_consensus.cc:515] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.061336 776 raft_consensus.cc:3060] T d424e3387698434f92e84eb1d43f5949 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.061744 2991 leader_election.cc:290] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:34.062952 713 leader_election.cc:343] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: feb19ea389af4ea6bb158a0e8aaeb094
I20251214 20:52:34.064874 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feb19ea389af4ea6bb158a0e8aaeb094" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.064934 2884 raft_consensus.cc:493] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.065096 908 raft_consensus.cc:2468] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:34.065068 2884 raft_consensus.cc:515] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.065717 2884 leader_election.cc:290] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.066315 2906 raft_consensus.cc:493] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.066452 2906 raft_consensus.cc:515] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.066983 2906 leader_election.cc:290] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.067725 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d171301941fa43908e05984c3b016007" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:34.068408 991 leader_election.cc:343] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.074736 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d171301941fa43908e05984c3b016007" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.075906 712 leader_election.cc:304] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.076104 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a000951f61948ac85fdd431f3d162c2" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.076262 908 raft_consensus.cc:2468] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
W20251214 20:52:34.077605 713 leader_election.cc:343] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 6a000951f61948ac85fdd431f3d162c2
W20251214 20:52:34.077593 989 leader_election.cc:343] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.077695 989 leader_election.cc:304] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.078059 2906 raft_consensus.cc:2749] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.078083 712 leader_election.cc:304] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.078529 2819 raft_consensus.cc:2804] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.078613 2819 raft_consensus.cc:493] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.078670 2819 raft_consensus.cc:3060] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.078691 2991 raft_consensus.cc:2804] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.078773 2991 raft_consensus.cc:493] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.078830 2991 raft_consensus.cc:3060] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.081545 2819 raft_consensus.cc:515] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.081806 2991 raft_consensus.cc:515] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.082219 2819 leader_election.cc:290] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.082520 3006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "e6713a20a82b434f8a8bc122113e26ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } } }
I20251214 20:52:34.082753 3006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad [sys.catalog]: This master's current role is: LEADER
I20251214 20:52:34.083438 2991 leader_election.cc:290] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.084085 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feb19ea389af4ea6bb158a0e8aaeb094" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:34.084182 2889 consensus_queue.cc:1048] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.084237 908 raft_consensus.cc:3060] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.084897 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a000951f61948ac85fdd431f3d162c2" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:34.085054 910 raft_consensus.cc:3060] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.087745 910 raft_consensus.cc:2468] T 6a000951f61948ac85fdd431f3d162c2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:34.087790 908 raft_consensus.cc:2468] T feb19ea389af4ea6bb158a0e8aaeb094 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
W20251214 20:52:34.088930 713 leader_election.cc:343] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 6a000951f61948ac85fdd431f3d162c2
W20251214 20:52:34.089123 713 leader_election.cc:343] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: feb19ea389af4ea6bb158a0e8aaeb094
W20251214 20:52:34.089768 848 consensus_peers.cc:597] T f11834cabdad445b8e53cd237d146282 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.092272 712 leader_election.cc:304] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.092728 712 leader_election.cc:304] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.092971 2884 raft_consensus.cc:2804] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.093128 2884 raft_consensus.cc:697] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:34.093403 2884 consensus_queue.cc:237] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.095777 2991 raft_consensus.cc:2804] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.096408 2991 raft_consensus.cc:697] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:34.096765 2991 consensus_queue.cc:237] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.098776 2906 raft_consensus.cc:493] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.098958 2906 raft_consensus.cc:515] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.099645 2906 leader_election.cc:290] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.100036 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd511a5dca4a2cbda81fbc255ce354" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.100426 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd511a5dca4a2cbda81fbc255ce354" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.100721 2819 raft_consensus.cc:493] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.100833 2819 raft_consensus.cc:515] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:34.100945 991 leader_election.cc:343] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.101431 2819 leader_election.cc:290] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:34.101609 989 leader_election.cc:343] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.101698 989 leader_election.cc:304] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.102108 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b69bb4d5979f492683fc13238a668b9f" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.102479 2906 raft_consensus.cc:2749] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:34.103412 713 leader_election.cc:343] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: b69bb4d5979f492683fc13238a668b9f
I20251214 20:52:34.103530 713 leader_election.cc:304] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.103901 3039 raft_consensus.cc:2749] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.113551 1203 tablet_bootstrap.cc:492] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.113739 1203 ts_tablet_manager.cc:1403] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.002s
I20251214 20:52:34.114550 1203 raft_consensus.cc:359] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.114753 1203 raft_consensus.cc:385] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.114817 1203 raft_consensus.cc:740] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.115051 1203 consensus_queue.cc:260] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.116196 1203 ts_tablet_manager.cc:1434] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:34.116628 1203 tablet_bootstrap.cc:492] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.116994 2786 raft_consensus.cc:493] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.117173 2786 raft_consensus.cc:515] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.117827 2786 leader_election.cc:290] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.118376 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "727e7b61ae3942098712720d344c5310" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.118594 1050 raft_consensus.cc:2468] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.118613 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "727e7b61ae3942098712720d344c5310" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.119022 851 leader_election.cc:304] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:34.119283 1203 tablet_bootstrap.cc:654] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.119609 2786 raft_consensus.cc:2804] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.119704 2786 raft_consensus.cc:493] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.119773 2786 raft_consensus.cc:3060] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.122433 2786 raft_consensus.cc:515] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:34.123577 848 leader_election.cc:343] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.124531 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "727e7b61ae3942098712720d344c5310" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:34.125031 848 leader_election.cc:343] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.125375 2786 leader_election.cc:290] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.126065 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "727e7b61ae3942098712720d344c5310" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.126374 1050 raft_consensus.cc:3060] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.128014 1203 tablet_bootstrap.cc:492] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.128166 1203 ts_tablet_manager.cc:1403] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20251214 20:52:34.129436 1203 raft_consensus.cc:359] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.129654 1203 raft_consensus.cc:385] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.129701 1203 raft_consensus.cc:740] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.129807 2946 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251214 20:52:34.129947 1050 raft_consensus.cc:2468] T 727e7b61ae3942098712720d344c5310 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.129947 1203 consensus_queue.cc:260] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.131404 1203 ts_tablet_manager.cc:1434] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:34.131778 1203 tablet_bootstrap.cc:492] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.134233 1203 tablet_bootstrap.cc:654] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.135619 851 leader_election.cc:304] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.135998 2786 raft_consensus.cc:2804] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.136116 2786 raft_consensus.cc:697] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.136359 2786 consensus_queue.cc:237] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:34.139691 848 consensus_peers.cc:597] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.140421 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_15c1781afe3449ef87ef3c655ccca0df [id=02c00037bc944c00b0124d72af62e19b]
I20251214 20:52:34.141062 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_f1e9c435093240e4b1c1eb95635c8e66 [id=03997e2eab154ef18b2e3774b5984764]
I20251214 20:52:34.141600 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]
I20251214 20:52:34.142022 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974]
I20251214 20:52:34.142474 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_eec6b34ea20f4ef096ef8df379a3bddb [id=05a65b055df144b0bd3e83d8baba49a0]
I20251214 20:52:34.142778 2991 raft_consensus.cc:493] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.142900 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_6838d68b72c042a293fd6c13096e6b78 [id=065328c700574078a1380f42b9848e2c]
I20251214 20:52:34.142983 2991 raft_consensus.cc:515] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:34.143877 2991 leader_election.cc:290] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.144670 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "492770a7b8b74ba49315f974cc54eedb" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.144899 908 raft_consensus.cc:2468] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:34.145355 712 leader_election.cc:304] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:34.145445 1203 tablet_bootstrap.cc:492] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.145650 1203 ts_tablet_manager.cc:1403] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
W20251214 20:52:34.145680 713 leader_election.cc:343] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 492770a7b8b74ba49315f974cc54eedb
I20251214 20:52:34.146052 2991 raft_consensus.cc:2804] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.146131 2991 raft_consensus.cc:493] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.146190 2991 raft_consensus.cc:3060] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.146677 1203 raft_consensus.cc:359] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.146874 1203 raft_consensus.cc:385] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.146919 1203 raft_consensus.cc:740] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.147205 1203 consensus_queue.cc:260] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.148321 1203 ts_tablet_manager.cc:1434] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251214 20:52:34.148839 2991 raft_consensus.cc:515] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:34.149016 1203 tablet_bootstrap.cc:492] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:34.150291 713 leader_election.cc:343] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 492770a7b8b74ba49315f974cc54eedb
I20251214 20:52:34.151337 1203 tablet_bootstrap.cc:654] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.151387 2991 leader_election.cc:290] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.152153 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "492770a7b8b74ba49315f974cc54eedb" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:34.152354 908 raft_consensus.cc:3060] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.154992 908 raft_consensus.cc:2468] T 492770a7b8b74ba49315f974cc54eedb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:34.156108 712 leader_election.cc:304] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.156646 2991 raft_consensus.cc:2804] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.156834 2991 raft_consensus.cc:697] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:34.157121 2991 consensus_queue.cc:237] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:34.168594 2906 raft_consensus.cc:493] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.168759 2906 raft_consensus.cc:515] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.169351 2906 leader_election.cc:290] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.170301 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e185d107619414a8bcc15ad496547dc" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.171358 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e185d107619414a8bcc15ad496547dc" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:34.172535 991 leader_election.cc:343] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:34.173871 989 leader_election.cc:343] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.174019 989 leader_election.cc:304] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.174629 2906 raft_consensus.cc:2749] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.177881 1203 tablet_bootstrap.cc:492] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.178073 1203 ts_tablet_manager.cc:1403] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.008s
I20251214 20:52:34.178961 1203 raft_consensus.cc:359] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.179208 1203 raft_consensus.cc:385] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.179277 1203 raft_consensus.cc:740] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.179512 1203 consensus_queue.cc:260] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.180454 1203 ts_tablet_manager.cc:1434] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251214 20:52:34.192786 2777 raft_consensus.cc:493] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.192966 2777 raft_consensus.cc:515] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.193594 2777 leader_election.cc:290] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.194211 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfe50a6219134431b0a025317932609b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.180776 1203 tablet_bootstrap.cc:492] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:34.194733 848 leader_election.cc:343] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.187374 2906 raft_consensus.cc:493] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.196066 2906 raft_consensus.cc:515] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.196254 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfe50a6219134431b0a025317932609b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.196499 1050 raft_consensus.cc:2468] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.196544 1203 tablet_bootstrap.cc:654] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.196671 2906 leader_election.cc:290] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.190853 2735 raft_consensus.cc:493] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.196897 2735 raft_consensus.cc:515] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.197302 2735 leader_election.cc:290] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.197670 851 leader_election.cc:304] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.198168 2889 raft_consensus.cc:2804] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.198239 2889 raft_consensus.cc:493] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.198292 2889 raft_consensus.cc:3060] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.200778 2889 raft_consensus.cc:515] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.201382 2889 leader_election.cc:290] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.202869 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfe50a6219134431b0a025317932609b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:34.203822 848 leader_election.cc:343] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.204046 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_1225d07e2e7a4aa1b60557b64de1f35e [id=07f7bb3503114ba3b0e1fe404826d582]
I20251214 20:52:34.204556 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]
I20251214 20:52:34.205003 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_176fae22b3bf4a159ac799ec6339ea16 [id=08bd7456e0db4786a4f2508d2709abea]
I20251214 20:52:34.204949 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76472ce9005b4bd2bbc51278816e8ae0" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.205044 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532c44785f204f988518c6fd628f76ca" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:34.205962 991 leader_election.cc:343] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.206068 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b289a25bf8174d9199c90212ce70271d [id=0a8ffa32e9ce4e3a8f9fa2afa4de69bb]
W20251214 20:52:34.206200 991 leader_election.cc:343] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.206676 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_93730665b1ab4686a0d3e54740fbddc8 [id=0b5581be22d345b3a0fd22c3dce2f207]
I20251214 20:52:34.207852 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532c44785f204f988518c6fd628f76ca" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.208133 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76472ce9005b4bd2bbc51278816e8ae0" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:34.208413 989 leader_election.cc:343] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.208531 989 leader_election.cc:304] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:34.208938 989 leader_election.cc:343] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.209036 989 leader_election.cc:304] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.209170 2735 raft_consensus.cc:2749] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.209365 2906 raft_consensus.cc:2749] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.217509 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfe50a6219134431b0a025317932609b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.219072 1051 raft_consensus.cc:3060] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.225315 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_d59e2b0d82384da68c1901ddaa419d00 [id=0c56f14b40ac442994e232aa348d64a2]
I20251214 20:52:34.226061 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_59b03580c6ff4f28baeb68e68994d7dc [id=0c71124d6937494cbded6debf9e9e4be]
I20251214 20:52:34.231650 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_adc1626f17674e3cab49c72818eec7d6 [id=0d173bfba2b343dbac6a02a5ff84c828]
I20251214 20:52:34.235623 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_3b0a7d868e794df9bf7ec67f8b299c35 [id=0e2f32942b8d49c68d9fd9212e0702aa]
I20251214 20:52:34.236390 1051 raft_consensus.cc:2468] T dfe50a6219134431b0a025317932609b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
W20251214 20:52:34.237099 848 consensus_peers.cc:597] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.238404 851 leader_election.cc:304] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.238819 1203 tablet_bootstrap.cc:492] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.238992 2890 raft_consensus.cc:2804] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.239048 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b6011f3e00584819944d165d653d06ac [id=103545908ce040858f00aed970b27479]
I20251214 20:52:34.239113 2890 raft_consensus.cc:697] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.239180 1203 ts_tablet_manager.cc:1403] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.058s user 0.017s sys 0.017s
I20251214 20:52:34.239436 2890 consensus_queue.cc:237] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.239698 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5cda3bedf2d24fc2b279c049146aa2c5 [id=10ef2e196edd432aa21fd7d31fa4366f]
I20251214 20:52:34.240058 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5de120dfaab44b84868db0d1f79e2cee [id=1158f8a25d2b4696a5f5ed2c3a426862]
I20251214 20:52:34.240392 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_00ec2fefc3d84a8b9f94bf9d194e2045 [id=12e7c17407a444dd8e0c0ad40fb89e28]
I20251214 20:52:34.240759 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_f4c54cf75bea445185ff40f822983fb5 [id=12f54ade07c6481298fe1a6783f719d4]
I20251214 20:52:34.240617 1203 raft_consensus.cc:359] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.240887 1203 raft_consensus.cc:385] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.240993 1203 raft_consensus.cc:740] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.241114 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ac56c984d6f5476cb187409e4e7575b6 [id=1350143e22c44ac59f5ac65a3c2f5763]
I20251214 20:52:34.241362 1203 consensus_queue.cc:260] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.242807 1203 ts_tablet_manager.cc:1434] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251214 20:52:34.244825 1206 tablet_bootstrap.cc:492] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.245129 1206 ts_tablet_manager.cc:1403] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.193s user 0.006s sys 0.049s
I20251214 20:52:34.246426 1206 raft_consensus.cc:359] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.246757 1206 raft_consensus.cc:385] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.246896 1206 raft_consensus.cc:740] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.247319 1206 consensus_queue.cc:260] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.248499 1206 ts_tablet_manager.cc:1434] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:34.251077 1203 tablet_bootstrap.cc:492] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.251718 1202 tablet_bootstrap.cc:492] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.251886 1202 ts_tablet_manager.cc:1403] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.199s user 0.000s sys 0.006s
I20251214 20:52:34.252547 1202 raft_consensus.cc:359] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.252712 1202 raft_consensus.cc:385] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.252767 1202 raft_consensus.cc:740] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.252975 1202 consensus_queue.cc:260] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.253583 1202 ts_tablet_manager.cc:1434] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:34.255189 1206 tablet_bootstrap.cc:492] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.255179 1202 tablet_bootstrap.cc:492] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.257148 1202 tablet_bootstrap.cc:654] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.257150 1206 tablet_bootstrap.cc:654] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.265048 1202 tablet_bootstrap.cc:492] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.265785 1202 ts_tablet_manager.cc:1403] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20251214 20:52:34.266050 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_4bb4b496a62d4c89b7b4e783023dd5cd [id=144f1eb0434a44f49e52b76ba687525c]
I20251214 20:52:34.266395 1203 tablet_bootstrap.cc:654] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.266532 1202 raft_consensus.cc:359] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.266695 1202 raft_consensus.cc:385] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.266747 1202 raft_consensus.cc:740] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.266997 1202 consensus_queue.cc:260] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.268368 776 raft_consensus.cc:3060] T c6a8863ace4541aea7463af24db27142 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.268775 1206 tablet_bootstrap.cc:492] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.268886 1206 ts_tablet_manager.cc:1403] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20251214 20:52:34.269563 1206 raft_consensus.cc:359] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.269718 1206 raft_consensus.cc:385] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.269759 1206 raft_consensus.cc:740] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.269974 1206 consensus_queue.cc:260] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.270560 1206 ts_tablet_manager.cc:1434] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:34.270807 1206 tablet_bootstrap.cc:492] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.272410 1206 tablet_bootstrap.cc:654] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.276755 1202 ts_tablet_manager.cc:1434] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251214 20:52:34.281365 1203 tablet_bootstrap.cc:492] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.281563 1203 ts_tablet_manager.cc:1403] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.003s
I20251214 20:52:34.282424 1203 raft_consensus.cc:359] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.282668 1203 raft_consensus.cc:385] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.282766 1203 raft_consensus.cc:740] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.283185 1203 consensus_queue.cc:260] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.284030 1203 ts_tablet_manager.cc:1434] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:34.284354 1203 tablet_bootstrap.cc:492] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.286106 1203 tablet_bootstrap.cc:654] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.289510 2890 raft_consensus.cc:493] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.291944 2890 raft_consensus.cc:515] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.292598 2890 leader_election.cc:290] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.293480 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "161fa219cc2f4c44bbcfbdfe9dce9a1f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.294226 2884 raft_consensus.cc:493] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.294354 2884 raft_consensus.cc:515] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:34.294871 2884 leader_election.cc:290] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:34.295750 848 leader_election.cc:343] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.296002 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1f4ea72c6274569a98cab6afdc55d55" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.296257 910 raft_consensus.cc:2468] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:34.297088 712 leader_election.cc:304] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:34.297044 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "161fa219cc2f4c44bbcfbdfe9dce9a1f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.297256 1050 raft_consensus.cc:2468] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.297519 2884 raft_consensus.cc:2804] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.297598 2884 raft_consensus.cc:493] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.297655 2884 raft_consensus.cc:3060] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:34.298233 713 leader_election.cc:343] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: b1f4ea72c6274569a98cab6afdc55d55
I20251214 20:52:34.298734 851 leader_election.cc:304] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.299098 2777 consensus_queue.cc:1048] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.299492 2777 raft_consensus.cc:2804] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.299554 2777 raft_consensus.cc:493] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.299603 2777 raft_consensus.cc:3060] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.302255 2777 raft_consensus.cc:515] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.302816 2777 leader_election.cc:290] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.303174 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_dce91ff3d9c74c7db1d400a24c4e550a [id=14f12b293a274ebeaf5bfefdff5b78cf]
I20251214 20:52:34.303648 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "161fa219cc2f4c44bbcfbdfe9dce9a1f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:34.304042 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_9098a11909c844dca6d4e80a284aa2e8 [id=1597e24e3fff4007ab66b44505c960d8]
W20251214 20:52:34.304214 848 leader_election.cc:343] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.304291 2777 raft_consensus.cc:493] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.304526 2777 raft_consensus.cc:515] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.305500 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "161fa219cc2f4c44bbcfbdfe9dce9a1f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.305595 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0ad1aa4193467db74cd707e7084b71" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.305610 2890 consensus_queue.cc:1048] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.305682 1050 raft_consensus.cc:3060] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:34.306066 848 leader_election.cc:343] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.306444 2777 leader_election.cc:290] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.308193 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c]
I20251214 20:52:34.308485 1050 raft_consensus.cc:2468] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.309576 851 leader_election.cc:304] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.309854 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0ad1aa4193467db74cd707e7084b71" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.310109 1050 raft_consensus.cc:2468] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.310312 2890 raft_consensus.cc:2804] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.310456 2890 raft_consensus.cc:697] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.310725 2890 consensus_queue.cc:237] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.312258 851 leader_election.cc:304] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.312619 2786 raft_consensus.cc:2804] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.312711 2786 raft_consensus.cc:493] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.312774 2786 raft_consensus.cc:3060] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.313908 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b2ac4ca26b6640a287e7f3b6974c2363 [id=15ed55ac8a0249bfbc08076265293bd6]
I20251214 20:52:34.315485 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_bdfed01cc4cd4458be399de18a738f45 [id=164c3a6291b644789ae5905faca45b10]
I20251214 20:52:34.315589 2786 raft_consensus.cc:515] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.316243 2786 leader_election.cc:290] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.317143 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0ad1aa4193467db74cd707e7084b71" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:34.318533 848 leader_election.cc:343] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.318668 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0ad1aa4193467db74cd707e7084b71" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.319429 1050 raft_consensus.cc:3060] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.319671 1206 tablet_bootstrap.cc:492] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.319818 1206 ts_tablet_manager.cc:1403] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.000s
I20251214 20:52:34.320523 1206 raft_consensus.cc:359] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.320655 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_bf1d2eb618b64a4e86cd121824fec0cb [id=1c14038af47c43c28ec8370fba0a14df]
I20251214 20:52:34.320688 1206 raft_consensus.cc:385] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.320741 1206 raft_consensus.cc:740] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.320922 1206 consensus_queue.cc:260] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.321121 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_0c936131512c4d99b3999d74fabf18d3 [id=1c5148c58b1c4fe39bac0697004ca484]
I20251214 20:52:34.321393 2884 raft_consensus.cc:515] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:34.321655 1206 ts_tablet_manager.cc:1434] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:34.321892 1206 tablet_bootstrap.cc:492] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.322404 2884 leader_election.cc:290] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.323580 1206 tablet_bootstrap.cc:654] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:34.325814 713 leader_election.cc:343] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: b1f4ea72c6274569a98cab6afdc55d55
I20251214 20:52:34.326568 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1f4ea72c6274569a98cab6afdc55d55" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:34.326819 910 raft_consensus.cc:3060] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.327607 1050 raft_consensus.cc:2468] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.328347 851 leader_election.cc:304] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.328698 2786 raft_consensus.cc:2804] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.328833 2786 raft_consensus.cc:697] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.282243 1202 tablet_bootstrap.cc:492] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.329098 2786 consensus_queue.cc:237] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.330790 1202 tablet_bootstrap.cc:654] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.330225 910 raft_consensus.cc:2468] T b1f4ea72c6274569a98cab6afdc55d55 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:34.331631 712 leader_election.cc:304] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.332171 2884 raft_consensus.cc:2804] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.332607 2884 raft_consensus.cc:697] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:34.332883 2884 consensus_queue.cc:237] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:34.334422 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_f8b17194060c4627a7770e45799e082b [id=1c9da7e7dc974945b2b5ccf304d1c1a8]
I20251214 20:52:34.335366 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_250b3222a83c44b08856444499fa612d [id=1ca50be529bd48398643cb6882958f77]
I20251214 20:52:34.336624 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_58bb2f3098084d73bafbd51cde268a21 [id=1e84b30c44fc49ba8a8b35c7e3252761]
I20251214 20:52:34.337152 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_2be03ca8d3964e4db04c188da2e59bd3 [id=1eeba8e8c0be4ac2b904a59363a89e99]
I20251214 20:52:34.338045 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_501d1ae22d314c82914be72476d1ee0d [id=1f1f307f926f4ce297cf3f9c5423e818]
I20251214 20:52:34.339347 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_c404f3a3b71f4899a233f864e78a7d13 [id=203681807a4b40b0a1f69387cfecff38]
I20251214 20:52:34.341228 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_94c723f6fee24f40a5787f7e5e92d03f [id=223e7226cfa94865a38476e84504a946]
I20251214 20:52:34.341377 768 raft_consensus.cc:3060] T 4f1d0e555eee4c56be271d14c34335bd P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.341776 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]
I20251214 20:52:34.342322 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_fc742d049909446bbdb1a0b0973734c0 [id=233d035cea634e6a82f2505aa9bf0e32]
I20251214 20:52:34.342981 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 [id=23efc0ee36244ca0a6ff0f2faeb00b1f]
I20251214 20:52:34.343878 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_c50d34acaf2e4ec8be9f4c60d90ec25c [id=2605ab6b89da49229adbc897734da196]
I20251214 20:52:34.344485 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_e7a205677f614917bc69ca73b5fb54da [id=262d9252040b4138ad42d1c05a402cd7]
I20251214 20:52:34.344945 2777 consensus_queue.cc:1048] T 4f1d0e555eee4c56be271d14c34335bd P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.347425 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_c96142a94af44f24bb7c6e51bf5e1f3e [id=296657ce558f4c599fc2013a71b7bda3]
I20251214 20:52:34.347944 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b14d897359c140bab6d04403fd59f31a [id=2a055be9028342be97f5c98f03f5a137]
I20251214 20:52:34.348938 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_040e46c43e8446f48a60e34f0892bf57 [id=2b6d1f218393414d868d37d390b35b58]
I20251214 20:52:34.351250 2889 consensus_queue.cc:1048] T d2a57409d4f54f17be792e6f3f17d7b7 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.360721 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_4278bf82042b47c2b4d0e799f115c52f [id=2c037ca94c8a4c908d6354f2aa802bf6]
I20251214 20:52:34.362623 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_581df19239f343f89fbe7e5c17d6d85c [id=2c6c7db9251948fe840e9fc0cc2076db]
I20251214 20:52:34.363597 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ebccdf7648384d44b9629a1b41509e68 [id=2e1e99c74ef5467190886342facb1a5b]
I20251214 20:52:34.364315 2906 raft_consensus.cc:493] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.364506 2906 raft_consensus.cc:515] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.365151 2906 leader_election.cc:290] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.365526 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_d9babf534d4b4411aa03906984198232 [id=2e2d38b971be42a1befe0d6995030970]
I20251214 20:52:34.365794 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d5920fdba794e5f864c595e7be278ab" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.366146 1206 tablet_bootstrap.cc:492] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.366168 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_7b56d13e748e405a821ea513405846fa [id=2e995b9d47b04d3d8f5a01482c57d0d2]
I20251214 20:52:34.366719 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_196d0f488e0b42aabd518511a4289dc3 [id=2f57c28c9c624a6ca1986c8831b85169]
W20251214 20:52:34.367048 991 leader_election.cc:343] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.367926 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_8923490b824a47139d8f739d65e4845b [id=2ffffc8a53e04583a2f93b287fe083bf]
I20251214 20:52:34.368678 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_cdf0b37254924fa18a712b2bd472fb46 [id=3322c0d4eb1a409da134030cd0a993fe]
I20251214 20:52:34.366853 2884 raft_consensus.cc:493] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.367389 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d5920fdba794e5f864c595e7be278ab" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.371502 2884 raft_consensus.cc:515] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.369982 2991 raft_consensus.cc:493] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.371788 2991 raft_consensus.cc:515] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:34.371829 989 leader_election.cc:343] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.371928 989 leader_election.cc:304] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.372284 2906 raft_consensus.cc:2749] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.372370 2991 leader_election.cc:290] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.372658 2777 consensus_queue.cc:1048] T 082ec70b8ee646b898d9201ad073d292 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.373204 1203 tablet_bootstrap.cc:492] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.373340 1203 ts_tablet_manager.cc:1403] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.011s
I20251214 20:52:34.373605 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65e79db74fa046698507c83416ee1669" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.374418 2884 leader_election.cc:290] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.374665 1206 ts_tablet_manager.cc:1403] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.012s
W20251214 20:52:34.375057 713 leader_election.cc:343] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: eb29fd8c85344ed8aa8d880e2ed240bc
I20251214 20:52:34.375245 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb29fd8c85344ed8aa8d880e2ed240bc" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.375434 910 raft_consensus.cc:2468] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:34.375897 712 leader_election.cc:304] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.376016 1206 raft_consensus.cc:359] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.376164 1206 raft_consensus.cc:385] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.376207 1206 raft_consensus.cc:740] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.376413 1206 consensus_queue.cc:260] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.376638 2991 raft_consensus.cc:2804] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.376695 2991 raft_consensus.cc:493] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.376747 2991 raft_consensus.cc:3060] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.379302 2991 raft_consensus.cc:515] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.379844 2991 leader_election.cc:290] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.380712 1202 tablet_bootstrap.cc:492] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.380841 1202 ts_tablet_manager.cc:1403] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.099s user 0.000s sys 0.004s
W20251214 20:52:34.381407 713 leader_election.cc:343] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: eb29fd8c85344ed8aa8d880e2ed240bc
I20251214 20:52:34.381489 1202 raft_consensus.cc:359] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.381624 1202 raft_consensus.cc:385] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.381680 1202 raft_consensus.cc:740] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.381851 1202 consensus_queue.cc:260] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.382364 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eb29fd8c85344ed8aa8d880e2ed240bc" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:34.382580 910 raft_consensus.cc:3060] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.384472 1202 ts_tablet_manager.cc:1434] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251214 20:52:34.385205 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
/tmp/dist-test-taskM_U6oP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42865
--webserver_interface=127.0.104.62
--webserver_port=44579
--builtin_ntp_servers=127.0.104.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251214 20:52:34.388970 910 raft_consensus.cc:2468] T eb29fd8c85344ed8aa8d880e2ed240bc P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
W20251214 20:52:34.389294 713 leader_election.cc:343] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 65e79db74fa046698507c83416ee1669
I20251214 20:52:34.389426 713 leader_election.cc:304] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.389782 2991 raft_consensus.cc:2749] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.389958 712 leader_election.cc:304] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.390282 1206 ts_tablet_manager.cc:1434] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251214 20:52:34.393240 1202 tablet_bootstrap.cc:492] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.394007 2991 raft_consensus.cc:2804] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.395020 1203 raft_consensus.cc:359] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.395224 1203 raft_consensus.cc:385] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.395278 1203 raft_consensus.cc:740] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.395442 2991 raft_consensus.cc:697] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:34.395599 1206 tablet_bootstrap.cc:492] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.395781 1203 consensus_queue.cc:260] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.395793 2991 consensus_queue.cc:237] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.397150 1203 ts_tablet_manager.cc:1434] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251214 20:52:34.397886 1206 tablet_bootstrap.cc:654] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:34.402733 848 consensus_peers.cc:597] T 140667afe42f42659a8cd38ed2517201 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.404142 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_52200ef9aa154f64ae25af61268f6d6b [id=35d3089ba6ef4f28ada7749dc8fb5ceb]
I20251214 20:52:34.406123 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_a5e4f40e603f4fd0a3d204f9c0298023 [id=36a5b0a44e8548e0a3c3f28975c4337d]
I20251214 20:52:34.408010 1203 tablet_bootstrap.cc:492] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.408028 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_8cd3b56154ec4d9fac4ed571a2287cfa [id=39df4da3b39545d3ba2c9345618ae532]
I20251214 20:52:34.409726 1203 tablet_bootstrap.cc:654] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.412256 1202 tablet_bootstrap.cc:654] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.415036 1203 tablet_bootstrap.cc:492] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.415246 1203 ts_tablet_manager.cc:1403] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20251214 20:52:34.418844 2906 raft_consensus.cc:493] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.418985 2906 raft_consensus.cc:515] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.419628 2906 leader_election.cc:290] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.420261 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7f06eded6c340c58101fcc644ebcf50" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.420814 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7f06eded6c340c58101fcc644ebcf50" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:34.421265 991 leader_election.cc:343] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:34.421504 989 leader_election.cc:343] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.421586 989 leader_election.cc:304] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.422009 2906 raft_consensus.cc:2749] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.423812 1206 tablet_bootstrap.cc:492] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.423996 1206 ts_tablet_manager.cc:1403] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.002s
I20251214 20:52:34.424646 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_bde8bb1617d3450f999e9a7960d96641 [id=3a3dca8b90264d949ac9bead0b0743fc]
I20251214 20:52:34.424757 1206 raft_consensus.cc:359] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.425817 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_d8bb40c2324c4c0eb7e52aa96e920813 [id=3b04d86591b14695a334b68b0572f371]
I20251214 20:52:34.427758 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_debb0f541d1b4ca3bc00c5e2c81c8428 [id=3b70fc138995441896f4ec7341fb69d4]
I20251214 20:52:34.428165 1203 raft_consensus.cc:359] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.428373 1203 raft_consensus.cc:385] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.428429 1203 raft_consensus.cc:740] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.428642 1203 consensus_queue.cc:260] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.428282 1206 raft_consensus.cc:385] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.429409 1206 raft_consensus.cc:740] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.429792 1206 consensus_queue.cc:260] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.430725 1206 ts_tablet_manager.cc:1434] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251214 20:52:34.435078 1206 tablet_bootstrap.cc:492] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.439770 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_0945dd902049447298aef35db32f33ba [id=3b87708d1c7b460d925f7da030a1a23f]
I20251214 20:52:34.440450 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_42e9fda7b7d14640a5804197bfb41683 [id=3bd3b6b8ee1f4d7abc6f888fc9d9004a]
I20251214 20:52:34.440623 1203 ts_tablet_manager.cc:1434] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251214 20:52:34.441128 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_cb1220aeec334a3c8aa15d0049c58d75 [id=3e833b0eaf3a4f519a1c1cfebaa0e969]
I20251214 20:52:34.441797 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_09270706a48c42fea27376b3ef3a0746 [id=3f15c09d26b04d519d92594446cca818]
I20251214 20:52:34.442034 2906 raft_consensus.cc:493] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.442229 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_58619bc0f4b54a208cabd270329ccf82 [id=3f4e5f296df7465b9c9ef0c86e7e2b76]
I20251214 20:52:34.442356 2906 raft_consensus.cc:515] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.442674 1206 tablet_bootstrap.cc:654] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.443152 2906 leader_election.cc:290] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.443886 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28d34670fdce4242a3feccf830a12880" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.444564 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90]
W20251214 20:52:34.444566 991 leader_election.cc:343] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.444952 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5a79c363f13a4849a6be94ea935e5b89 [id=4093d9a107134e4b90970ddecc931769]
I20251214 20:52:34.445567 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28d34670fdce4242a3feccf830a12880" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.446022 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_7219830d8fb040debc2388f49091b802 [id=40d3f605109b43828c497df2bf4eef65]
W20251214 20:52:34.446169 989 leader_election.cc:343] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.446285 989 leader_election.cc:304] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.446462 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5b82f109385a42b0a04a5272bc5f0156 [id=40f4f5bacfea4828880fc526be3a06cb]
I20251214 20:52:34.446694 1202 tablet_bootstrap.cc:492] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.446842 1202 ts_tablet_manager.cc:1403] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20251214 20:52:34.446862 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]
I20251214 20:52:34.446892 2906 raft_consensus.cc:2749] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.447292 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_fcafcf3c6ddb4457aea41320f7bf0b74 [id=4329d01e7f6147d0adad53fa86369b17]
I20251214 20:52:34.447656 1202 raft_consensus.cc:359] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.447832 1202 raft_consensus.cc:385] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.447890 1202 raft_consensus.cc:740] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.447774 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_8f897cb4591a4d5fadbf2b1b7ee7408d [id=4390cb960d564bb5bc7e0cefa5bf3004]
I20251214 20:52:34.448064 1202 consensus_queue.cc:260] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.448529 1202 ts_tablet_manager.cc:1434] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:34.449213 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_873207e85b954e21b3d98ad4c02a0248 [id=451df68541d848278bfeda2dfce0ba7f]
I20251214 20:52:34.450794 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_6588026e2d974db9b72d2b81ce8310cb [id=483610ffc7db4e848d0756374015974f]
I20251214 20:52:34.454887 1202 tablet_bootstrap.cc:492] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.456580 1202 tablet_bootstrap.cc:654] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.456897 2889 raft_consensus.cc:493] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.457033 2889 raft_consensus.cc:515] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.457849 2889 leader_election.cc:290] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.458575 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292a28b12df246a8bc855d81c530746e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.458832 1052 raft_consensus.cc:2468] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.458839 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292a28b12df246a8bc855d81c530746e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.459304 851 leader_election.cc:304] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:34.459633 848 leader_election.cc:343] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.459959 2889 raft_consensus.cc:2804] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.460047 2889 raft_consensus.cc:493] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.460120 2889 raft_consensus.cc:3060] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.460318 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_9ff099dcfac14636a9dcaa70207c86a6 [id=4a1816b57ff84e459394e59b683d8ec6]
I20251214 20:52:34.461575 1203 tablet_bootstrap.cc:492] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.463510 1203 tablet_bootstrap.cc:654] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.465052 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_81b328890d6748ea8a95ac3cd696c9dd [id=4a964e807c8a4161bf9fa9cb732ba21d]
I20251214 20:52:34.465098 2889 raft_consensus.cc:515] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.466125 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292a28b12df246a8bc855d81c530746e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:34.466197 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_9698b9206b934e4db2decdf9c0180838 [id=4b3f6c8d8cce4f7a96b2c677df6469ba]
I20251214 20:52:34.466130 2889 leader_election.cc:290] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.466176 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "292a28b12df246a8bc855d81c530746e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.466308 1052 raft_consensus.cc:3060] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:34.466784 848 leader_election.cc:343] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.467806 1206 tablet_bootstrap.cc:492] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.468024 1206 ts_tablet_manager.cc:1403] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.004s
I20251214 20:52:34.468942 1206 raft_consensus.cc:359] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.469135 1052 raft_consensus.cc:2468] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.469208 1206 raft_consensus.cc:385] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.469373 1206 raft_consensus.cc:740] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.469643 851 leader_election.cc:304] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.469662 1206 consensus_queue.cc:260] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.470119 2889 raft_consensus.cc:2804] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.470657 2889 raft_consensus.cc:697] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.471105 2889 consensus_queue.cc:237] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.471858 1203 tablet_bootstrap.cc:492] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.471964 1203 ts_tablet_manager.cc:1403] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20251214 20:52:34.473098 1206 ts_tablet_manager.cc:1434] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:34.473483 1206 tablet_bootstrap.cc:492] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.483325 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_4fdc1b366f04459b8b083811e9740ae1 [id=4ba4d4bb0a2d48b4aa7b956cafdb26a7]
I20251214 20:52:34.483901 1206 tablet_bootstrap.cc:654] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.484665 1203 raft_consensus.cc:359] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.484869 1203 raft_consensus.cc:385] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.484925 1203 raft_consensus.cc:740] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.485138 1203 consensus_queue.cc:260] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.486236 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_8ae98ffc7458410786e7844521f3c5c1 [id=4c6dacf190dd4c7aa5b4e056bf7dfd8c]
I20251214 20:52:34.487185 1203 ts_tablet_manager.cc:1434] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251214 20:52:34.487564 1203 tablet_bootstrap.cc:492] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.487506 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b72c21c5b00d430ea93eff8bf8bcb228 [id=4d54f97a2cde4143a185eb4af35bb07d]
I20251214 20:52:34.488417 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]
I20251214 20:52:34.489100 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]
I20251214 20:52:34.489596 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_4cb054a62c404f1a952f39810b6d12f8 [id=515eec002ab94ec396c6bc70dae4d996]
I20251214 20:52:34.489777 1203 tablet_bootstrap.cc:654] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.493935 768 raft_consensus.cc:3060] T 65e79db74fa046698507c83416ee1669 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.495082 2991 raft_consensus.cc:493] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.495286 2991 raft_consensus.cc:515] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.495956 2991 leader_election.cc:290] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.496728 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f945814952b14131ba4e0c67a403a680" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.497717 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ca7b705bd1754ce19fbd7cda96364054 [id=525aea874ba84a65af4b2cbbed3fbf9e]
I20251214 20:52:34.497925 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f945814952b14131ba4e0c67a403a680" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.498607 713 leader_election.cc:304] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.498986 2884 raft_consensus.cc:2749] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.502362 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_12d421ed905d41eead75b19eb8a793b6 [id=5272550343144d4baaba07e97ebaa187]
I20251214 20:52:34.505759 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]
I20251214 20:52:34.506049 1202 tablet_bootstrap.cc:492] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.506233 1202 ts_tablet_manager.cc:1403] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.001s
I20251214 20:52:34.506806 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_1be39fb576af4b2b9e2606b495619863 [id=54e0182cd3b9461aa593a11aa7709002]
I20251214 20:52:34.506919 1202 raft_consensus.cc:359] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.507102 1202 raft_consensus.cc:385] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.507197 1202 raft_consensus.cc:740] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.507380 1202 consensus_queue.cc:260] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.507889 1202 ts_tablet_manager.cc:1434] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:34.508543 1206 tablet_bootstrap.cc:492] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.508702 1206 ts_tablet_manager.cc:1403] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.000s
I20251214 20:52:34.509509 1206 raft_consensus.cc:359] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.509687 1206 raft_consensus.cc:385] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.509740 1206 raft_consensus.cc:740] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.509986 1206 consensus_queue.cc:260] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.510686 1206 ts_tablet_manager.cc:1434] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:34.510982 1206 tablet_bootstrap.cc:492] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.513988 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_0fb8f39742d245ac9d20fc7d6bf7e7c3 [id=552c8b213ba64d4bbdd3363cdc6625ff]
I20251214 20:52:34.514551 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_7d6d418a1acb404f908e9ee299cef249 [id=55fc2a6c5e514513904682dfde510d9f]
I20251214 20:52:34.515203 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_0f409000b9514206a08757040e81a6a9 [id=5692c4a553044543ab275cff0f7a0566]
I20251214 20:52:34.516014 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_17c9057bb25b4cce9b1072a93f6ea38a [id=5727ded20a714c31be1ba62f2dfb21e4]
I20251214 20:52:34.516412 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_d6683a16a95f4274b82ff74dc7e73597 [id=573bdb65191b4f09b43190844f2b0008]
I20251214 20:52:34.516993 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_3474e4ca80964e3b8b7b66e1fde4815f [id=586a58c36e9a4d4494f4538ebcb9ce36]
I20251214 20:52:34.517577 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_c3bedc14bbf14c5988f9c60ceacf9d95 [id=58ace009e5fc4eab93d8babc6330f58e]
I20251214 20:52:34.518168 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_87794c3eaa7c4545acddac9ad4b4bc93 [id=59e0f0da4a1447a6957080f16c820e19]
I20251214 20:52:34.518575 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_98b0ed640a93417ba49359781348b105 [id=5addd19777df4b968708a41c59a19651]
I20251214 20:52:34.519186 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_6fedc617de7e4cd985c616534bda854a [id=5c2a10eb6dca403a94f387331ec8c3d4]
I20251214 20:52:34.519687 1202 tablet_bootstrap.cc:492] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.520253 1206 tablet_bootstrap.cc:654] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.521426 1202 tablet_bootstrap.cc:654] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.526005 1202 tablet_bootstrap.cc:492] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.526158 1202 ts_tablet_manager.cc:1403] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20251214 20:52:34.526826 1202 raft_consensus.cc:359] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.527000 1202 raft_consensus.cc:385] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.527056 1202 raft_consensus.cc:740] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.527338 1202 consensus_queue.cc:260] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.527856 1202 ts_tablet_manager.cc:1434] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:34.528141 1202 tablet_bootstrap.cc:492] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.529963 1202 tablet_bootstrap.cc:654] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.532714 2884 raft_consensus.cc:493] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.532892 2884 raft_consensus.cc:515] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.533617 2884 leader_election.cc:290] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.534406 1202 tablet_bootstrap.cc:492] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.534555 1202 ts_tablet_manager.cc:1403] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20251214 20:52:34.536154 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_a54f22962823421ea5ab3033e0d9add0 [id=5db835c4244847e4969597e99df50099]
I20251214 20:52:34.536777 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c1893b3761c49ceb7cd71b3bd9f2cb4" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.536948 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c1893b3761c49ceb7cd71b3bd9f2cb4" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.537644 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_55f303d33ec84a269232fc1ba8f2d182 [id=5ded9fe825ae4308b5bd4b21c16634c8]
I20251214 20:52:34.537947 1202 raft_consensus.cc:359] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.538093 1202 raft_consensus.cc:385] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.538122 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_47baf32513e6487fbd4ffde9bcba97ca [id=5ea1e541e5454f1ca96338573fda5705]
I20251214 20:52:34.538141 1202 raft_consensus.cc:740] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.538376 1202 consensus_queue.cc:260] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.538746 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_02bb72c146234757b420555977074934 [id=5f186d0854254ce287edb0ccb77ee6b7]
I20251214 20:52:34.538944 1202 ts_tablet_manager.cc:1434] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:34.539212 1202 tablet_bootstrap.cc:492] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.539206 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ac4df504ee284227bfd4ba2a23eaff65 [id=5f69f1d7797b4fff9030914e710a8fb8]
I20251214 20:52:34.539657 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_cddd59a230e04595bffdf9e0a0ed1471 [id=6094596f6c064c5bb1e20aa2b8767b9f]
I20251214 20:52:34.539724 1206 tablet_bootstrap.cc:492] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.539973 1206 ts_tablet_manager.cc:1403] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.002s
I20251214 20:52:34.540351 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_76f578444d71410a93c3f78ff50b7951 [id=6157817e96d642d09f9941ad7f7e82ee]
I20251214 20:52:34.540727 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_f7d781182d07441a824d887e07286dc3 [id=6170186dc9cc4f4e93394530eb7c3a56]
I20251214 20:52:34.540879 1202 tablet_bootstrap.cc:654] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.541451 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_da9e4432a9254052ace01a09e425f97b [id=6203767083704e8b818e367287cfb130]
I20251214 20:52:34.541936 1206 raft_consensus.cc:359] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.542348 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_407ddd842d284ecd87a391f1b54a2b5a [id=66d124c2a8dd4cc188a832eff1dbe9ed]
I20251214 20:52:34.542320 1206 raft_consensus.cc:385] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.542481 1206 raft_consensus.cc:740] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.542822 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_eaf150707b2442f1ae6f94ac564c6b87 [id=67577dd83c4045c992f1c574bd71edfe]
I20251214 20:52:34.543015 1206 consensus_queue.cc:260] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.545290 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_eaeae6c30719483681416f2d216d1209 [id=68304e9495f44caab2e47d46335ab30f]
I20251214 20:52:34.545368 1206 ts_tablet_manager.cc:1434] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251214 20:52:34.545827 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_c3aa8e3c31cb41dc8281962e7b5c46e1 [id=687eb544b4114bad8d2deb76388e2022]
I20251214 20:52:34.546473 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_80d7626996ca40979563d484ae097f08 [id=694ef81722d948638bbec8847c69d75a]
I20251214 20:52:34.546486 712 leader_election.cc:304] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.546983 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_53bf327c948f4cebbe6096266ac301f8 [id=6a279e57008549c593df838211cc9cc1]
I20251214 20:52:34.547170 2884 raft_consensus.cc:2749] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.547626 2889 raft_consensus.cc:493] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.547752 2889 raft_consensus.cc:515] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.548202 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7]
I20251214 20:52:34.548312 2889 leader_election.cc:290] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.549038 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7aa508b11944aa8e268741c1998a9e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.549326 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7aa508b11944aa8e268741c1998a9e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.549559 1056 raft_consensus.cc:2468] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.549996 851 leader_election.cc:304] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:34.550276 2890 raft_consensus.cc:2804] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.550376 2890 raft_consensus.cc:493] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.550455 2890 raft_consensus.cc:3060] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.550733 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_acd41f528acd483cb2c37531c45283d6 [id=6cdbb231eb7e4d01b6023731b39f16a4]
W20251214 20:52:34.552709 848 leader_election.cc:343] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.553248 2890 raft_consensus.cc:515] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.553956 2890 leader_election.cc:290] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.554375 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7aa508b11944aa8e268741c1998a9e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.554584 1056 raft_consensus.cc:3060] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.554710 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7aa508b11944aa8e268741c1998a9e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:34.555269 848 leader_election.cc:343] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.557281 1056 raft_consensus.cc:2468] T 1d7aa508b11944aa8e268741c1998a9e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.557711 851 leader_election.cc:304] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.557869 1202 tablet_bootstrap.cc:492] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.558039 1202 ts_tablet_manager.cc:1403] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20251214 20:52:34.558120 2786 raft_consensus.cc:2804] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.558290 2786 raft_consensus.cc:697] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.558573 2786 consensus_queue.cc:237] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.558921 1202 raft_consensus.cc:359] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.553840 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_91b2da75e7104e0f95ed470c749f843f [id=70aba52f026e439f8bf0b12373148fca]
I20251214 20:52:34.561259 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4]
I20251214 20:52:34.561910 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ae25c010c04548198844cd6f5f3a5693 [id=7159a40b641e406d89e8a11b2b1eba82]
I20251214 20:52:34.562328 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_09f08f5ffa814302a74b268170036a24 [id=71cefadd096649a8aca97ece15ff1bb9]
I20251214 20:52:34.562748 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5126357c033645d1ae06a909124ab640 [id=72c5a8ce76cb46d89b7a563f8b16abc3]
I20251214 20:52:34.563370 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_70cb422b9a3d439cba4cddcdd9b31c73 [id=74eb42fa8016492cad785dfa4d0088c9]
I20251214 20:52:34.563767 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_08a9cb7994804ae586d3a7e3de4adb7d [id=74f23f7675974c82953d757dd563179a]
I20251214 20:52:34.564388 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_4649d8b3e43c42a492723fadd4217e4e [id=75cebf8d0aba4aa48c1be51b0abd6b38]
I20251214 20:52:34.564993 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_47682ee6f46147558b5033379c2fe733 [id=788e2409150b408996fcc9e70c026bd0]
I20251214 20:52:34.565780 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b9ef85d4768d42d28d635a7c3eac5f4c [id=7be3dc7e56824beb9f13c242645bdff4]
I20251214 20:52:34.566188 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_90e94bab31e54a9cb8b567622aa4e37d [id=7c3d3542270e449ca954d068bece3491]
I20251214 20:52:34.566606 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ae43058f9a39467b9919a9e1c335602f [id=7c91475975864cac93bfeb4d13173f1b]
I20251214 20:52:34.572083 1203 tablet_bootstrap.cc:492] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.572252 1203 ts_tablet_manager.cc:1403] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.006s
I20251214 20:52:34.573911 1206 tablet_bootstrap.cc:492] T f11834cabdad445b8e53cd237d146282 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.574440 1202 raft_consensus.cc:385] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.574599 1202 raft_consensus.cc:740] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.574896 1202 consensus_queue.cc:260] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.575814 1202 ts_tablet_manager.cc:1434] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251214 20:52:34.576483 1202 tablet_bootstrap.cc:492] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.578675 1202 tablet_bootstrap.cc:654] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.579210 2884 raft_consensus.cc:493] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.579380 2884 raft_consensus.cc:515] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.579824 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_cca1629ea2974eddb3a2fab0ae9f4500 [id=7ceb70a9b9b0458692720cd351d9a3c6]
I20251214 20:52:34.579998 2884 leader_election.cc:290] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.580895 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b47ca1a03944cf1847d3578856dafa7" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.581079 910 raft_consensus.cc:2468] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:34.581158 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_f9499a2114f44353aaf65b35ed56f119 [id=7dd81c38dde94205a84d6f7841d2bb73]
W20251214 20:52:34.581547 713 leader_election.cc:343] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 3b47ca1a03944cf1847d3578856dafa7
I20251214 20:52:34.581784 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_68c79df4e9e0446f81c8500bcfa423a6 [id=7e9bbbe22677488d9509dc201b2d7234]
I20251214 20:52:34.581887 712 leader_election.cc:304] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.582234 2884 raft_consensus.cc:2804] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.582297 2884 raft_consensus.cc:493] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.582350 2884 raft_consensus.cc:3060] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.583995 1203 raft_consensus.cc:359] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.584540 1203 raft_consensus.cc:385] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.584590 1203 raft_consensus.cc:740] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.585076 1203 consensus_queue.cc:260] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.585999 2884 raft_consensus.cc:515] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.586084 1206 tablet_bootstrap.cc:654] T f11834cabdad445b8e53cd237d146282 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:34.587461 713 leader_election.cc:343] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 3b47ca1a03944cf1847d3578856dafa7
I20251214 20:52:34.587700 2884 leader_election.cc:290] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.587877 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_624b9386a78b44e994cde78539ccedc4 [id=7ea13a9516b340dabbb4916b419b8c03]
I20251214 20:52:34.588212 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b47ca1a03944cf1847d3578856dafa7" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:34.588411 910 raft_consensus.cc:3060] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.590898 910 raft_consensus.cc:2468] T 3b47ca1a03944cf1847d3578856dafa7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:34.592097 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_2ca54bf356e24a1ebca6f4a155b83044 [id=7f407f5abea945498cf939a23b0f9319]
I20251214 20:52:34.593873 1203 ts_tablet_manager.cc:1434] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251214 20:52:34.593924 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_da61d1abe3f24498977b6cc43d2ed4df [id=800f51296fe14db2985b673976f6ec6c]
I20251214 20:52:34.594151 1203 tablet_bootstrap.cc:492] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.594336 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_e552103bd07e4f68b25576d6044d8861 [id=80a316658e6d494387628a03406f6480]
I20251214 20:52:34.594755 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_180ac2f089ba44298206726a47570c5b [id=81609d89fec84b2f9b79ca8986447b65]
I20251214 20:52:34.595880 1202 tablet_bootstrap.cc:492] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.596160 1202 ts_tablet_manager.cc:1403] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20251214 20:52:34.597419 1202 raft_consensus.cc:359] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.597841 1202 raft_consensus.cc:385] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.597960 1202 raft_consensus.cc:740] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.598234 1202 consensus_queue.cc:260] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.599313 1202 ts_tablet_manager.cc:1434] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:34.599674 1202 tablet_bootstrap.cc:492] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.598026 712 leader_election.cc:304] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.598687 773 raft_consensus.cc:3060] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.602238 2884 raft_consensus.cc:2804] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.602828 2884 raft_consensus.cc:697] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:34.604226 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_87f9c82a6a89490391b5c78001639d2b [id=8203de75aba94c3f8561f18adaaa825e]
I20251214 20:52:34.604883 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_2541d743bd8945a2b0d984cdf911d6ba [id=825f8141b3ca4a1983f6548c66239504]
I20251214 20:52:34.605504 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_4d5d73b55b6245bf8594a4fc79f02adc [id=8316131458234291a566634324832ffb]
I20251214 20:52:34.606473 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_db687aab85f2411f8433eb14724da9ac [id=83ef50f71fb34ecda09e60f00f8ec8a7]
I20251214 20:52:34.606863 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_3bc5f340978c453190d06f9e7a87f65c [id=87835c51994146ffa7f67ece6f50ff40]
I20251214 20:52:34.607283 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_13e60d7ade27439bbcab22ee125a50ba [id=8789a018b1c24bc3956fd11aea52b20e]
I20251214 20:52:34.607623 1202 tablet_bootstrap.cc:654] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.616648 2884 consensus_queue.cc:237] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.619187 1206 tablet_bootstrap.cc:492] T f11834cabdad445b8e53cd237d146282 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.619663 1206 ts_tablet_manager.cc:1403] T f11834cabdad445b8e53cd237d146282 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.005s
I20251214 20:52:34.621250 1206 raft_consensus.cc:359] T f11834cabdad445b8e53cd237d146282 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.621743 1206 raft_consensus.cc:385] T f11834cabdad445b8e53cd237d146282 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.621934 1206 raft_consensus.cc:740] T f11834cabdad445b8e53cd237d146282 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.622859 1206 consensus_queue.cc:260] T f11834cabdad445b8e53cd237d146282 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.623730 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_a07fe706644d4c63b966f0751b757642 [id=8845cef019b44676a83a3173c5df0bb3]
I20251214 20:52:34.623914 2991 raft_consensus.cc:493] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.624054 2991 raft_consensus.cc:515] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.624647 2991 leader_election.cc:290] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.625769 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7353593bda4fceaa389c393ddc19c6" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.626021 910 raft_consensus.cc:2468] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
W20251214 20:52:34.626171 713 leader_election.cc:343] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: fc7353593bda4fceaa389c393ddc19c6
I20251214 20:52:34.626523 712 leader_election.cc:304] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.626665 1202 tablet_bootstrap.cc:492] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.626822 1202 ts_tablet_manager.cc:1403] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20251214 20:52:34.626950 3039 raft_consensus.cc:2804] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.627048 3039 raft_consensus.cc:493] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.627166 3039 raft_consensus.cc:3060] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.627697 1202 raft_consensus.cc:359] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.627859 1202 raft_consensus.cc:385] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.627918 1202 raft_consensus.cc:740] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.627959 1206 ts_tablet_manager.cc:1434] T f11834cabdad445b8e53cd237d146282 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251214 20:52:34.628279 1206 tablet_bootstrap.cc:492] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.630448 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ec8c8f9185be4470a6580c4e141e2227 [id=8872bfac95c64cbba1ccb84d1d9f2483]
I20251214 20:52:34.630924 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_0a325ff1f4634ce7ada593df8f5a9fd6 [id=88a6486d16184397acd4ef205ca46a7a]
I20251214 20:52:34.631392 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b87980e2da6a4052b109938fcf0527d8 [id=88d5807cc76646e3a2962b5d36413894]
I20251214 20:52:34.631815 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_80391b5a5aa9425794dfb7997c659009 [id=893edd42fa04409b9d8e4a4e46d38ebc]
I20251214 20:52:34.632122 1203 tablet_bootstrap.cc:654] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.632285 1206 tablet_bootstrap.cc:654] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.632578 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5c7eb24742b84515af287c257637e51b [id=8acc06886693413682b246f25970fa5b]
I20251214 20:52:34.633832 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_7306395aef6240c39e095e0c8815130b [id=8cd62da3d57c4be4b5f10ace49d2e148]
I20251214 20:52:34.635707 3039 raft_consensus.cc:515] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.635942 2991 raft_consensus.cc:493] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.636142 2991 raft_consensus.cc:515] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.636540 3039 leader_election.cc:290] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.636617 2991 leader_election.cc:290] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.637087 1202 consensus_queue.cc:260] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.637794 1202 ts_tablet_manager.cc:1434] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251214 20:52:34.639549 1202 tablet_bootstrap.cc:492] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.640902 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc7353593bda4fceaa389c393ddc19c6" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:34.641175 910 raft_consensus.cc:3060] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:34.642019 713 leader_election.cc:343] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: e6ebfc373fbf41b1920c271a15302484
W20251214 20:52:34.642257 713 leader_election.cc:343] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: fc7353593bda4fceaa389c393ddc19c6
I20251214 20:52:34.643298 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6ebfc373fbf41b1920c271a15302484" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.643896 712 leader_election.cc:304] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.644356 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_fc1d7cb6f8544f6285ec8792deb1da74 [id=8ce51bbd64ff4e1789fade77a6b0c13f]
I20251214 20:52:34.645128 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_e3feb6d6cacb48dca8455eacf4cfb84b [id=8f720514e13b45689817a9811fb624d9]
W20251214 20:52:34.645382 848 consensus_peers.cc:597] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.645567 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_1f214ce7548c43af80a74e233714ca0d [id=8f83a24aa5834c4b8c8bec3f0a8ac29e]
I20251214 20:52:34.645607 2991 raft_consensus.cc:2749] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.646404 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_529fb2317e754462b7ae49d9304e5402 [id=9187b59d5c7949169b104f586e7edad7]
I20251214 20:52:34.648985 910 raft_consensus.cc:2468] T fc7353593bda4fceaa389c393ddc19c6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:34.651889 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_70e73cbd7f9141f3ab482647b6118d52 [id=91a01b7a64464481b8ebe6331c0746da]
I20251214 20:52:34.652956 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_679d53b1472341ef8612ca53ac1ebda2 [id=91f3b4dd176348a5aa38c6bdc417918f]
I20251214 20:52:34.654562 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_398f36b3acb440d88213e38f63740ca5 [id=954b3b4acdd94043b2608438d00483e7]
I20251214 20:52:34.655752 1203 tablet_bootstrap.cc:492] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.655967 1203 ts_tablet_manager.cc:1403] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.006s
I20251214 20:52:34.654351 1202 tablet_bootstrap.cc:654] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.658102 1203 raft_consensus.cc:359] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.658355 1203 raft_consensus.cc:385] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.658362 2991 raft_consensus.cc:493] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.658509 1203 raft_consensus.cc:740] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.658547 2991 raft_consensus.cc:515] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.658740 1203 consensus_queue.cc:260] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.659257 1203 ts_tablet_manager.cc:1434] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:34.659273 2991 leader_election.cc:290] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.659581 1203 tablet_bootstrap.cc:492] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.661566 1203 tablet_bootstrap.cc:654] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.665004 2786 raft_consensus.cc:493] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.665195 773 raft_consensus.cc:3060] T b69bb4d5979f492683fc13238a668b9f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.665194 2786 raft_consensus.cc:515] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.665880 2786 leader_election.cc:290] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.666688 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7f06eded6c340c58101fcc644ebcf50" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:34.667313 848 leader_election.cc:343] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.667322 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5de6830aaec645f2992e4cfaceb20c27 [id=9553b46856c94003a293135c814a7df7]
I20251214 20:52:34.667436 712 leader_election.cc:304] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.667943 2991 raft_consensus.cc:2804] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.668094 2991 raft_consensus.cc:697] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:34.668401 2991 consensus_queue.cc:237] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.668856 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_1052ac11eee9483aa13db090acff5b7a [id=978e62482f114f2b99e9707aaa9a5e32]
I20251214 20:52:34.669579 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d]
I20251214 20:52:34.670320 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7f06eded6c340c58101fcc644ebcf50" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.670519 1052 raft_consensus.cc:2468] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.670658 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_531eb871a88242c5b91423a2c6abc3b6 [id=9afdb569356648bb84089b1738dfc359]
I20251214 20:52:34.670850 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe24e4f4e74040249068505058c15622" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.670939 851 leader_election.cc:304] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.671077 910 raft_consensus.cc:2468] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:34.671362 2786 raft_consensus.cc:2804] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.671480 2786 raft_consensus.cc:493] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.671564 2786 raft_consensus.cc:3060] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:34.672240 713 leader_election.cc:343] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: fe24e4f4e74040249068505058c15622
I20251214 20:52:34.672688 712 leader_election.cc:304] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.672963 2884 raft_consensus.cc:2804] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.673060 2884 raft_consensus.cc:493] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.673154 2884 raft_consensus.cc:3060] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.674456 2786 raft_consensus.cc:515] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.675819 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7f06eded6c340c58101fcc644ebcf50" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:34.676008 2884 raft_consensus.cc:515] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:34.676743 848 leader_election.cc:343] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.677464 2884 leader_election.cc:290] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.678171 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe24e4f4e74040249068505058c15622" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
W20251214 20:52:34.678256 713 leader_election.cc:343] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: fe24e4f4e74040249068505058c15622
I20251214 20:52:34.678366 910 raft_consensus.cc:3060] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.680841 910 raft_consensus.cc:2468] T fe24e4f4e74040249068505058c15622 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:34.681321 712 leader_election.cc:304] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.681794 2884 raft_consensus.cc:2804] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.683383 1203 tablet_bootstrap.cc:492] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.684973 1203 ts_tablet_manager.cc:1403] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20251214 20:52:34.685936 1203 raft_consensus.cc:359] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.691263 1203 raft_consensus.cc:385] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.692214 1203 raft_consensus.cc:740] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.692765 1203 consensus_queue.cc:260] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:34.693001 713 consensus_peers.cc:597] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e1835374cfa48939d72476870accc3b. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.693830 1203 ts_tablet_manager.cc:1434] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:34.694013 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7f06eded6c340c58101fcc644ebcf50" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.694192 1203 tablet_bootstrap.cc:492] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.694226 1052 raft_consensus.cc:3060] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.693720 2786 leader_election.cc:290] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.697018 1052 raft_consensus.cc:2468] T f7f06eded6c340c58101fcc644ebcf50 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.698108 1203 tablet_bootstrap.cc:654] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.700023 851 leader_election.cc:304] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.700490 2889 raft_consensus.cc:493] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.700644 2889 raft_consensus.cc:515] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.701334 2889 leader_election.cc:290] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.701493 2889 raft_consensus.cc:2804] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.701608 2889 raft_consensus.cc:697] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.701848 2889 consensus_queue.cc:237] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.703867 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e11723c042a4637839f1b13bed0935a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.703967 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e11723c042a4637839f1b13bed0935a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.704136 1052 raft_consensus.cc:2468] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:34.704684 848 leader_election.cc:343] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.705217 851 leader_election.cc:304] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.705634 2786 raft_consensus.cc:2804] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.705729 2786 raft_consensus.cc:493] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.705816 2786 raft_consensus.cc:3060] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.706329 2906 consensus_queue.cc:1048] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.706936 1203 tablet_bootstrap.cc:492] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.707602 1203 ts_tablet_manager.cc:1403] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20251214 20:52:34.708606 1203 raft_consensus.cc:359] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.710996 2786 raft_consensus.cc:515] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.712101 1203 raft_consensus.cc:385] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.712222 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e11723c042a4637839f1b13bed0935a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.712373 1203 raft_consensus.cc:740] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.712440 1052 raft_consensus.cc:3060] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.712836 1203 consensus_queue.cc:260] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.714905 1052 raft_consensus.cc:2468] T 8e11723c042a4637839f1b13bed0935a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
W20251214 20:52:34.715386 989 consensus_peers.cc:597] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.716645 851 leader_election.cc:304] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:34.716806 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e11723c042a4637839f1b13bed0935a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:34.717000 2889 raft_consensus.cc:2804] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.717123 2889 raft_consensus.cc:697] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
W20251214 20:52:34.717594 848 leader_election.cc:343] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.717794 1203 ts_tablet_manager.cc:1434] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251214 20:52:34.717371 2889 consensus_queue.cc:237] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.719185 1203 tablet_bootstrap.cc:492] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.727137 1203 tablet_bootstrap.cc:654] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.731731 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_c6c0496d455a4db68c05c0ff8b06493d [id=9d1473df58ee486aa6a55893aca9d54c]
W20251214 20:52:34.734936 848 consensus_peers.cc:597] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.738408 2786 leader_election.cc:290] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.738504 2889 consensus_queue.cc:1048] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.741365 1202 tablet_bootstrap.cc:492] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.741616 1202 ts_tablet_manager.cc:1403] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.102s user 0.017s sys 0.007s
I20251214 20:52:34.742772 2786 raft_consensus.cc:493] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.742949 2786 raft_consensus.cc:515] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.742666 1202 raft_consensus.cc:359] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.743328 1202 raft_consensus.cc:385] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.744300 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76472ce9005b4bd2bbc51278816e8ae0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.744787 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76472ce9005b4bd2bbc51278816e8ae0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.745024 1056 raft_consensus.cc:2468] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.743455 1202 raft_consensus.cc:740] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.744344 2786 leader_election.cc:290] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:34.745211 848 leader_election.cc:343] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.745497 2786 raft_consensus.cc:493] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.745529 851 leader_election.cc:304] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.745663 2786 raft_consensus.cc:515] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.745916 2777 raft_consensus.cc:2804] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.745950 2884 raft_consensus.cc:697] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:34.746039 2777 raft_consensus.cc:493] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.746134 2777 raft_consensus.cc:3060] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.746241 2884 consensus_queue.cc:237] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.746730 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2688515bd1441fbad7516f8343ca775" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:34.747231 848 leader_election.cc:343] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.747745 1202 consensus_queue.cc:260] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.748946 2777 raft_consensus.cc:515] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.749341 1202 ts_tablet_manager.cc:1434] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251214 20:52:34.749518 2777 leader_election.cc:290] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.750190 2786 leader_election.cc:290] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.751003 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76472ce9005b4bd2bbc51278816e8ae0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.751195 1051 raft_consensus.cc:3060] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.753844 1051 raft_consensus.cc:2468] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.754411 2777 raft_consensus.cc:493] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.754627 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2688515bd1441fbad7516f8343ca775" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.754654 2777 raft_consensus.cc:515] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.754861 1056 raft_consensus.cc:2468] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.755261 2777 leader_election.cc:290] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.755318 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76472ce9005b4bd2bbc51278816e8ae0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:34.756062 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a07d94d5a431488489ee9fbcc113a9d7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:34.756738 848 leader_election.cc:343] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:34.756959 848 leader_election.cc:343] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.758391 2906 raft_consensus.cc:493] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.758574 851 leader_election.cc:304] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.758579 2906 raft_consensus.cc:515] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.758975 851 leader_election.cc:304] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.759291 2906 leader_election.cc:290] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.759330 2786 raft_consensus.cc:2804] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.759488 2950 raft_consensus.cc:2804] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.759550 2950 raft_consensus.cc:493] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.759613 2950 raft_consensus.cc:3060] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.759805 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a07d94d5a431488489ee9fbcc113a9d7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.759994 1056 raft_consensus.cc:2468] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.760730 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "812642cfcff940f899ef9d3644909ec4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:34.761389 991 leader_election.cc:343] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.762167 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "812642cfcff940f899ef9d3644909ec4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.762315 2950 raft_consensus.cc:515] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.762907 2950 leader_election.cc:290] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:34.762897 989 leader_election.cc:343] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.763021 989 leader_election.cc:304] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.763545 2906 raft_consensus.cc:2749] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.764243 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2688515bd1441fbad7516f8343ca775" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:34.764815 848 leader_election.cc:343] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.765147 2786 raft_consensus.cc:697] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.765504 2786 consensus_queue.cc:237] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.770923 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_7694eee14c2346a9a71802bdf0897011 [id=9f14a0a266b947559b2173b20ead5ccd]
I20251214 20:52:34.773847 1202 tablet_bootstrap.cc:492] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.777237 1202 tablet_bootstrap.cc:654] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.778168 1206 tablet_bootstrap.cc:492] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.779868 1206 ts_tablet_manager.cc:1403] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.151s user 0.016s sys 0.016s
I20251214 20:52:34.784090 1202 tablet_bootstrap.cc:492] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.779935 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_40e2218a9f58420e9a09ccfbda122485 [id=9fa32d9e7b7c49baa67e36e0eb996e2e]
I20251214 20:52:34.784883 1202 ts_tablet_manager.cc:1403] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20251214 20:52:34.785948 1202 raft_consensus.cc:359] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.787426 1202 raft_consensus.cc:385] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.787539 1202 raft_consensus.cc:740] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.787801 1202 consensus_queue.cc:260] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.788146 851 leader_election.cc:304] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.790222 2950 raft_consensus.cc:2804] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.790321 2950 raft_consensus.cc:493] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.790380 2950 raft_consensus.cc:3060] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.791352 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b5f628ed069a4b19996c9b579d7678bc [id=a03c04d94997449bac7447ed634577d4]
W20251214 20:52:34.792309 848 consensus_peers.cc:597] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.789464 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2688515bd1441fbad7516f8343ca775" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.793143 1052 raft_consensus.cc:3060] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.793365 1202 ts_tablet_manager.cc:1434] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251214 20:52:34.793411 1203 tablet_bootstrap.cc:492] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.793534 1203 ts_tablet_manager.cc:1403] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.000s
I20251214 20:52:34.792816 1206 raft_consensus.cc:359] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.794498 1203 raft_consensus.cc:359] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.794729 1203 raft_consensus.cc:385] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.794811 1203 raft_consensus.cc:740] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.794762 1206 raft_consensus.cc:385] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.795056 1203 consensus_queue.cc:260] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.795953 1203 ts_tablet_manager.cc:1434] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:34.796139 1052 raft_consensus.cc:2468] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.796331 1203 tablet_bootstrap.cc:492] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.795629 1206 raft_consensus.cc:740] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.798604 1203 tablet_bootstrap.cc:654] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.799790 1202 tablet_bootstrap.cc:492] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.799894 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_7fbc660486544d9fb711dcdca0660237 [id=a20045f442604c45a67b9b06f1eb4610]
I20251214 20:52:34.799018 1206 consensus_queue.cc:260] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.802165 2950 raft_consensus.cc:515] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.803227 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a07d94d5a431488489ee9fbcc113a9d7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:34.802829 2950 leader_election.cc:290] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.808348 1206 ts_tablet_manager.cc:1434] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.024s user 0.000s sys 0.003s
I20251214 20:52:34.808699 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_493f60b9d8024509856ed934d379f829 [id=a40a329d98674364aa3794d3a07fdeb4]
I20251214 20:52:34.805285 1202 tablet_bootstrap.cc:654] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:34.803817 848 leader_election.cc:343] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.811287 2889 raft_consensus.cc:493] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.811359 1206 tablet_bootstrap.cc:492] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:34.811551 2889 raft_consensus.cc:515] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.805807 851 leader_election.cc:304] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.812752 2890 raft_consensus.cc:2804] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.812788 2991 consensus_queue.cc:1048] T 9e1835374cfa48939d72476870accc3b P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.813243 2890 raft_consensus.cc:697] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.813668 2890 consensus_queue.cc:237] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.813964 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3915abc53b084854912a8e4119eefc7e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.814465 1203 tablet_bootstrap.cc:492] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.814651 1203 ts_tablet_manager.cc:1403] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.002s
W20251214 20:52:34.815099 848 leader_election.cc:343] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.815482 1203 raft_consensus.cc:359] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.815680 1203 raft_consensus.cc:385] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.815757 1203 raft_consensus.cc:740] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.815913 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a07d94d5a431488489ee9fbcc113a9d7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.816069 1203 consensus_queue.cc:260] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.816213 1056 raft_consensus.cc:3060] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:34.817924 848 consensus_peers.cc:597] T 9f04336d2c84439b9003df19d3541273 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.818010 1203 ts_tablet_manager.cc:1434] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251214 20:52:34.818581 2889 leader_election.cc:290] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.819783 1203 tablet_bootstrap.cc:492] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.819738 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_e125108df3c94234bedbdf467db81f28 [id=a47338e44a9e4642baa8407af70be735]
I20251214 20:52:34.820415 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3915abc53b084854912a8e4119eefc7e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.821074 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_eba09dc20b824904b171bc39be41ea65 [id=a4ef8163c0a741f290495ef9015ae6cb]
I20251214 20:52:34.821620 1052 raft_consensus.cc:2468] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.822157 1056 raft_consensus.cc:2468] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.823907 1203 tablet_bootstrap.cc:654] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.824630 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_912c9fc9392c4d99aa8485af7917eb85 [id=a56279b96c2d45af90277c7636192936]
I20251214 20:52:34.827239 1206 tablet_bootstrap.cc:654] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.827284 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_183ef00a0ed14492bb36f72ed1c73aa2 [id=a61d5b8118774bb2b806d755ab98da86]
I20251214 20:52:34.828208 3108 raft_consensus.cc:493] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.828423 3108 raft_consensus.cc:515] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.829483 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eb1006b0ba440dcba2f9de04dca2702" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.829839 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5cffb2d16a4847caa5cf6a2c4c238da8 [id=a8b3bbe26fc14b4bbc8a78c3d1f9df6d]
W20251214 20:52:34.830013 848 leader_election.cc:343] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.831593 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_dd6d020171b24861a33a63f13b128050 [id=aa336c07501b4a99afc12338d7556623]
I20251214 20:52:34.832635 1203 tablet_bootstrap.cc:492] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.836498 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_1dc40537749846a8a77948884d646257 [id=ae41e2897b3a4deda0dacb28baa72adf]
I20251214 20:52:34.838390 1203 ts_tablet_manager.cc:1403] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20251214 20:52:34.839385 1203 raft_consensus.cc:359] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.839742 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_4abdf07a08814d65acc20b6c32bfd415 [id=aeb3160d155949699e454f389fd07aa3]
I20251214 20:52:34.840458 1203 raft_consensus.cc:385] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.840652 1203 raft_consensus.cc:740] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.841041 1203 consensus_queue.cc:260] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.842185 1203 ts_tablet_manager.cc:1434] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251214 20:52:34.842828 1203 tablet_bootstrap.cc:492] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.845650 1203 tablet_bootstrap.cc:654] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.846143 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_9060872cac6842548b9f1f80d16dbe19 [id=b0f347532cc84fcca5892f5a2443321c]
I20251214 20:52:34.846812 768 raft_consensus.cc:3060] T e6ebfc373fbf41b1920c271a15302484 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.847818 3108 leader_election.cc:290] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.854094 1203 tablet_bootstrap.cc:492] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.854692 1203 ts_tablet_manager.cc:1403] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20251214 20:52:34.859424 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eb1006b0ba440dcba2f9de04dca2702" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.859603 1052 raft_consensus.cc:2468] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.859899 1202 tablet_bootstrap.cc:492] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:34.860073 1202 ts_tablet_manager.cc:1403] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.001s
I20251214 20:52:34.861016 1202 raft_consensus.cc:359] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.861218 1202 raft_consensus.cc:385] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.863014 851 leader_election.cc:304] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.866231 2889 raft_consensus.cc:2804] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.866359 2889 raft_consensus.cc:493] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.866470 2889 raft_consensus.cc:3060] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.866499 851 leader_election.cc:304] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.867157 3111 raft_consensus.cc:2804] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.867261 3111 raft_consensus.cc:493] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.867367 3111 raft_consensus.cc:3060] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.869969 3111 raft_consensus.cc:515] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.871675 1203 raft_consensus.cc:359] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.873456 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eb1006b0ba440dcba2f9de04dca2702" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:34.874095 848 leader_election.cc:343] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.870237 1202 raft_consensus.cc:740] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.874580 1202 consensus_queue.cc:260] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.876786 1202 ts_tablet_manager.cc:1434] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20251214 20:52:34.878228 2884 raft_consensus.cc:493] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.878469 2884 raft_consensus.cc:515] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.879477 2884 leader_election.cc:290] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.879907 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "082ec70b8ee646b898d9201ad073d292" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.875115 2906 raft_consensus.cc:493] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.882304 2906 raft_consensus.cc:515] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.882906 2906 leader_election.cc:290] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.879693 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "082ec70b8ee646b898d9201ad073d292" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.883772 713 leader_election.cc:304] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:34.884624 2884 raft_consensus.cc:2749] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:34.884958 1203 raft_consensus.cc:385] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.885171 1203 raft_consensus.cc:740] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.885468 1203 consensus_queue.cc:260] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.884476 1202 tablet_bootstrap.cc:492] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:34.886144 3111 leader_election.cc:290] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.885821 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eb1006b0ba440dcba2f9de04dca2702" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.886307 1052 raft_consensus.cc:3060] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.889063 1052 raft_consensus.cc:2468] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.889590 851 leader_election.cc:304] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.890852 1203 ts_tablet_manager.cc:1434] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20251214 20:52:34.891373 1203 tablet_bootstrap.cc:492] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.893441 2906 raft_consensus.cc:493] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.893649 2906 raft_consensus.cc:515] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.894264 2906 leader_election.cc:290] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.893478 1203 tablet_bootstrap.cc:654] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:34.895748 2950 raft_consensus.cc:493] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.896056 2950 raft_consensus.cc:515] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.896795 2950 leader_election.cc:290] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.897610 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e59b80ee919b4e1393fe317ed0e3a3d2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:34.898207 848 leader_election.cc:343] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.884464 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f342dafae1d04be3ba38f7c0e4f5d058" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:34.901193 991 leader_election.cc:343] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.901825 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4643effa71d4d18850913a59ae1c75d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.902158 1202 tablet_bootstrap.cc:654] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:34.902285 991 leader_election.cc:343] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.902623 2786 raft_consensus.cc:2804] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.902743 2786 raft_consensus.cc:697] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.903002 2786 consensus_queue.cc:237] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.904453 773 raft_consensus.cc:3060] T f945814952b14131ba4e0c67a403a680 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.904928 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e59b80ee919b4e1393fe317ed0e3a3d2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.905099 1052 raft_consensus.cc:2468] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.905822 851 leader_election.cc:304] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.906369 2950 raft_consensus.cc:2804] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.906455 2950 raft_consensus.cc:493] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.906513 2950 raft_consensus.cc:3060] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.910192 2950 raft_consensus.cc:515] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.910846 3108 consensus_queue.cc:1048] T 6eca34627bbc4a3f9915be72f3128370 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.911528 2950 leader_election.cc:290] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.911551 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e59b80ee919b4e1393fe317ed0e3a3d2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:34.912439 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e59b80ee919b4e1393fe317ed0e3a3d2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.912643 1050 raft_consensus.cc:3060] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.917085 1050 raft_consensus.cc:2468] T e59b80ee919b4e1393fe317ed0e3a3d2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
W20251214 20:52:34.916352 848 leader_election.cc:343] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.918694 851 leader_election.cc:304] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.919165 2950 raft_consensus.cc:2804] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.920379 2950 raft_consensus.cc:697] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.920787 2889 raft_consensus.cc:515] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.920756 2950 consensus_queue.cc:237] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.921397 2889 leader_election.cc:290] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.921890 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3915abc53b084854912a8e4119eefc7e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:34.922530 848 leader_election.cc:343] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.923187 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3915abc53b084854912a8e4119eefc7e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:34.923427 1050 raft_consensus.cc:3060] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.926467 1050 raft_consensus.cc:2468] T 3915abc53b084854912a8e4119eefc7e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:34.927100 851 leader_election.cc:304] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.927577 1206 tablet_bootstrap.cc:492] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:34.927858 1206 ts_tablet_manager.cc:1403] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.117s user 0.000s sys 0.008s
I20251214 20:52:34.928907 2889 raft_consensus.cc:2804] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.929004 2884 consensus_queue.cc:1048] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:34.929306 2889 raft_consensus.cc:697] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.929819 2889 consensus_queue.cc:237] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.935827 1203 tablet_bootstrap.cc:492] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.935999 1203 ts_tablet_manager.cc:1403] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.001s
I20251214 20:52:34.942353 1203 raft_consensus.cc:359] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.943758 1203 raft_consensus.cc:385] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.943836 1203 raft_consensus.cc:740] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.947665 1203 consensus_queue.cc:260] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.948426 1203 ts_tablet_manager.cc:1434] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20251214 20:52:34.949061 1203 tablet_bootstrap.cc:492] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:34.951567 1203 tablet_bootstrap.cc:654] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:34.956216 713 consensus_peers.cc:597] T feb19ea389af4ea6bb158a0e8aaeb094 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: feb19ea389af4ea6bb158a0e8aaeb094. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:34.960209 2906 raft_consensus.cc:493] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.960410 2906 raft_consensus.cc:515] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.961021 2906 leader_election.cc:290] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.961573 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd1161eebe884fe6b347bf517269c047" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:34.962054 991 leader_election.cc:343] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.964684 3039 raft_consensus.cc:493] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.964843 3039 raft_consensus.cc:515] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.965525 3039 leader_election.cc:290] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:34.966660 713 leader_election.cc:343] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: a0976ffcc649461fac74e20b5e402275
I20251214 20:52:34.971323 851 leader_election.cc:304] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.977914 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0976ffcc649461fac74e20b5e402275" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.978307 910 raft_consensus.cc:2468] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:34.978796 712 leader_election.cc:304] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:34.979447 3039 raft_consensus.cc:2804] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.979544 3039 raft_consensus.cc:493] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.979607 3039 raft_consensus.cc:3060] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:34.980839 2950 raft_consensus.cc:2804] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:34.981065 2950 raft_consensus.cc:697] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:34.981354 2950 consensus_queue.cc:237] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.981526 1203 tablet_bootstrap.cc:492] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:34.981681 1203 ts_tablet_manager.cc:1403] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.004s
I20251214 20:52:34.982371 1203 raft_consensus.cc:359] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.982525 1203 raft_consensus.cc:385] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:34.982424 3039 raft_consensus.cc:515] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.982568 1203 raft_consensus.cc:740] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:34.982748 1203 consensus_queue.cc:260] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.983040 3039 leader_election.cc:290] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.984352 2786 raft_consensus.cc:493] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.984501 2786 raft_consensus.cc:515] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:34.985010 2786 leader_election.cc:290] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:34.985850 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0976ffcc649461fac74e20b5e402275" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:34.985916 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd511a5dca4a2cbda81fbc255ce354" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:34.986029 910 raft_consensus.cc:3060] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:34.986593 848 leader_election.cc:343] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:34.986965 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd511a5dca4a2cbda81fbc255ce354" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:34.987228 1050 raft_consensus.cc:2468] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:34.988817 910 raft_consensus.cc:2468] T a0976ffcc649461fac74e20b5e402275 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:34.989060 2906 raft_consensus.cc:493] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.989226 2906 raft_consensus.cc:515] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.989859 2906 leader_election.cc:290] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:34.991684 1203 ts_tablet_manager.cc:1434] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251214 20:52:34.994473 3119 raft_consensus.cc:493] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:34.994908 3119 raft_consensus.cc:515] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:34.989485 851 leader_election.cc:304] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:34.991081 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d745583012b4465090d85c75470e0bc2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:34.998291 2786 raft_consensus.cc:2804] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:34.998390 2786 raft_consensus.cc:493] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:34.998446 2786 raft_consensus.cc:3060] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:34.998759 991 leader_election.cc:343] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.001482 2786 raft_consensus.cc:515] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.002293 2786 leader_election.cc:290] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.003147 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd511a5dca4a2cbda81fbc255ce354" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:35.003867 848 leader_election.cc:343] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.005069 712 leader_election.cc:304] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:35.005527 3039 raft_consensus.cc:2804] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.005673 3039 raft_consensus.cc:697] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:35.006176 3039 consensus_queue.cc:237] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:35.007506 713 leader_election.cc:343] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: a0976ffcc649461fac74e20b5e402275
I20251214 20:52:35.009323 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dcd511a5dca4a2cbda81fbc255ce354" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.010035 1056 raft_consensus.cc:3060] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.009788 2786 consensus_queue.cc:1048] T c8e33ba1149a44c294885d76f676160a P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:35.010550 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 129782us
I20251214 20:52:35.011586 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f342dafae1d04be3ba38f7c0e4f5d058" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.012693 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd1161eebe884fe6b347bf517269c047" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.012775 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d745583012b4465090d85c75470e0bc2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.012470 2786 consensus_queue.cc:1048] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251214 20:52:35.013746 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4643effa71d4d18850913a59ae1c75d" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.013742 2786 consensus_queue.cc:1048] T 79e621f29c01448399d22615284fccd4 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:35.016616 989 leader_election.cc:343] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.016782 989 leader_election.cc:304] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.020926 1203 tablet_bootstrap.cc:492] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:35.022871 1203 tablet_bootstrap.cc:654] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.023020 1206 raft_consensus.cc:359] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.027165 1056 raft_consensus.cc:2468] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.028302 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_25630f29e4904aaf86f8f65078bcf81e [id=b1ee89b437fb42a4b5e2c3d386ee0284]
I20251214 20:52:35.028842 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ee7a207364d74e6b81d6a72e57cacbd5 [id=b251a30885f6435fbfa430b6b50aee7a]
I20251214 20:52:35.029291 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_d050172b06964a17a220a95fb84d1081 [id=b2e03510ce354e1686aad6c081d52903]
I20251214 20:52:35.029933 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8]
I20251214 20:52:35.030550 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_d61784eed0b84f8caa485dbe70cdc366 [id=b5622661ea9f40f7ac11f44ae92a5194]
I20251214 20:52:35.023411 1206 raft_consensus.cc:385] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.031666 1206 raft_consensus.cc:740] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.034853 1202 tablet_bootstrap.cc:492] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.035418 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc8d725e6ac440899fc1cde2f7b546aa" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.038616 1202 ts_tablet_manager.cc:1403] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.154s user 0.002s sys 0.009s
I20251214 20:52:35.030766 2906 raft_consensus.cc:493] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:35.039402 991 leader_election.cc:343] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.039438 2906 raft_consensus.cc:515] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.040356 2906 leader_election.cc:290] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.041208 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf9e65040af4fbeb701619fe27edfb6" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:35.041831 991 leader_election.cc:343] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.042497 773 raft_consensus.cc:3060] T 082ec70b8ee646b898d9201ad073d292 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.043809 2906 raft_consensus.cc:2749] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.043838 851 leader_election.cc:304] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.044302 2786 raft_consensus.cc:2804] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.044433 2786 raft_consensus.cc:697] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.044801 2786 consensus_queue.cc:237] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:35.045250 989 leader_election.cc:343] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.046667 989 leader_election.cc:304] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.047500 2906 raft_consensus.cc:2749] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:35.048285 989 leader_election.cc:343] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.048877 1206 consensus_queue.cc:260] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.049847 989 leader_election.cc:304] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.050134 2906 raft_consensus.cc:2749] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:35.050585 989 leader_election.cc:343] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.051672 989 leader_election.cc:304] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.051949 2906 raft_consensus.cc:2749] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.053998 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc8d725e6ac440899fc1cde2f7b546aa" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.034698 3119 leader_election.cc:290] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.054876 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf9e65040af4fbeb701619fe27edfb6" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.054790 1202 raft_consensus.cc:359] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.055018 1202 raft_consensus.cc:385] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.055069 1202 raft_consensus.cc:740] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.055306 1202 consensus_queue.cc:260] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:35.055685 989 leader_election.cc:343] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.055862 989 leader_election.cc:304] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:35.056259 989 leader_election.cc:343] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.056392 989 leader_election.cc:304] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.056548 3119 raft_consensus.cc:2749] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.056818 2906 raft_consensus.cc:2749] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.058437 1202 ts_tablet_manager.cc:1434] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251214 20:52:35.057376 2950 raft_consensus.cc:493] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.058720 2950 raft_consensus.cc:515] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.059793 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "566c6e4761d5486d964e276d151e32b0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.060336 848 leader_election.cc:343] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.061748 2950 leader_election.cc:290] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:35.062240 848 consensus_peers.cc:597] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.062638 1206 ts_tablet_manager.cc:1434] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.134s user 0.003s sys 0.000s
I20251214 20:52:35.063222 1206 tablet_bootstrap.cc:492] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:35.063704 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "566c6e4761d5486d964e276d151e32b0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.063907 1051 raft_consensus.cc:2468] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.064535 3119 raft_consensus.cc:493] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.065052 3119 raft_consensus.cc:515] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.065300 851 leader_election.cc:304] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.065604 2890 raft_consensus.cc:2804] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.065693 2890 raft_consensus.cc:493] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.065769 2890 raft_consensus.cc:3060] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.068595 2890 raft_consensus.cc:515] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.069284 2890 leader_election.cc:290] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.071715 1202 tablet_bootstrap.cc:492] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:35.073961 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "566c6e4761d5486d964e276d151e32b0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.074273 1051 raft_consensus.cc:3060] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.077025 1206 tablet_bootstrap.cc:654] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.077787 1051 raft_consensus.cc:2468] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.078136 2889 raft_consensus.cc:493] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.078325 2889 raft_consensus.cc:515] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.079059 2889 leader_election.cc:290] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.079505 851 leader_election.cc:304] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:35.079656 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fd5566a34e2a82009cc9ce6bdd37" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.079828 2889 raft_consensus.cc:2804] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.080011 1052 raft_consensus.cc:2468] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.080065 2889 raft_consensus.cc:697] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.080447 2889 consensus_queue.cc:237] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.083280 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_65d3039202a645e899e5f752e1e3db42 [id=b6bebe3ab7e2449dafc61fdda3e3c38c]
I20251214 20:52:35.088617 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "566c6e4761d5486d964e276d151e32b0" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.089309 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fd5566a34e2a82009cc9ce6bdd37" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.089939 768 raft_consensus.cc:3060] T f11834cabdad445b8e53cd237d146282 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:35.093385 848 leader_election.cc:343] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:35.093858 848 leader_election.cc:343] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.095239 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_7041fb3e6d4040f08fd9a554a30e315d [id=b6e9928c0c2845fdb544b4b449aeb113]
I20251214 20:52:35.097774 1203 tablet_bootstrap.cc:492] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.098392 851 leader_election.cc:304] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.098878 3108 raft_consensus.cc:2804] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.098971 3108 raft_consensus.cc:493] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.099152 3108 raft_consensus.cc:3060] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.099283 1203 ts_tablet_manager.cc:1403] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.011s
I20251214 20:52:35.102355 3108 raft_consensus.cc:515] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.102967 1203 raft_consensus.cc:359] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.103956 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eff8a23d48c43e2bdbfb34f4bab6edf" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.104195 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fd5566a34e2a82009cc9ce6bdd37" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.104315 3108 leader_election.cc:290] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:35.104805 848 leader_election.cc:343] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:35.106364 713 consensus_peers.cc:597] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 274185a4115b458e8e9a376bdbcc870b. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.106530 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eff8a23d48c43e2bdbfb34f4bab6edf" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.104475 991 leader_election.cc:343] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.107285 1203 raft_consensus.cc:385] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.107646 1203 raft_consensus.cc:740] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.106971 3119 leader_election.cc:290] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:35.108956 989 leader_election.cc:343] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.109504 989 leader_election.cc:304] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.109186 1203 consensus_queue.cc:260] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.111511 2906 raft_consensus.cc:2749] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.112584 2786 consensus_queue.cc:1048] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.114879 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_f6c14f9b22304e8aafbe1afab658ae71 [id=b6efd20d2ef54f799f88337bfc4f7980]
I20251214 20:52:35.115937 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_78f00e8dd81a485cae8269bac0d08866 [id=bc52dc01429e446b9726bc62fd9b24b3]
W20251214 20:52:35.117936 848 consensus_peers.cc:597] T 80098b7c24824d129b8f25daf2115460 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.120132 1203 ts_tablet_manager.cc:1434] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20251214 20:52:35.123386 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fd5566a34e2a82009cc9ce6bdd37" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.123786 1051 raft_consensus.cc:3060] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.124182 1203 tablet_bootstrap.cc:492] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:35.132455 3039 consensus_queue.cc:1048] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.133522 1203 tablet_bootstrap.cc:654] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.135195 1206 tablet_bootstrap.cc:492] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:35.136559 1206 ts_tablet_manager.cc:1403] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.000s
I20251214 20:52:35.137797 1206 raft_consensus.cc:359] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.137996 1206 raft_consensus.cc:385] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.138048 1206 raft_consensus.cc:740] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.138281 1206 consensus_queue.cc:260] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.138813 1206 ts_tablet_manager.cc:1434] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:35.139168 1206 tablet_bootstrap.cc:492] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:35.140379 1051 raft_consensus.cc:2468] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.140872 3111 raft_consensus.cc:493] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.141007 1206 tablet_bootstrap.cc:654] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.141165 3111 raft_consensus.cc:515] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.141980 3111 leader_election.cc:290] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.143019 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e4dc66681ef48479ae3cb95b091564a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.143762 848 consensus_peers.cc:597] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:35.144016 848 leader_election.cc:343] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.146874 1202 tablet_bootstrap.cc:654] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.150593 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_6cfa43d70d2f426fb58f8ef32476c3ac [id=be1be47d388347e3a6b7a80105688be3]
I20251214 20:52:35.154350 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741]
I20251214 20:52:35.156985 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_a1cc2f56735544ad8ae3425ad4d00fdb [id=bec715d329a74c7a8092f8e9c7f9c341]
I20251214 20:52:35.159560 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_27222912e9214b879d8a30975aba5688 [id=c0729ac770394d7b87ef36e53f85c151]
I20251214 20:52:35.163929 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_fd97b3b527e240fbb6d62d3c33059904 [id=c1ab5c38b5ee4129a8bccf874894ddcf]
I20251214 20:52:35.165870 3039 consensus_queue.cc:1048] T 274185a4115b458e8e9a376bdbcc870b P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.167548 1206 tablet_bootstrap.cc:492] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:35.167922 1206 ts_tablet_manager.cc:1403] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20251214 20:52:35.167984 3039 raft_consensus.cc:493] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.168179 3039 raft_consensus.cc:515] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.168810 3039 leader_election.cc:290] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.169971 3039 consensus_queue.cc:1048] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.145255 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e4dc66681ef48479ae3cb95b091564a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.174037 1052 raft_consensus.cc:2468] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.169548 2906 raft_consensus.cc:493] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.174527 2906 raft_consensus.cc:515] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.174768 851 leader_election.cc:304] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.175238 3141 raft_consensus.cc:2804] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.175310 3141 raft_consensus.cc:493] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.175375 3141 raft_consensus.cc:3060] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.175366 2906 leader_election.cc:290] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.175989 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147c985fa7c94f7a9af57917bf17b29f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.176178 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147c985fa7c94f7a9af57917bf17b29f" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.170601 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "140667afe42f42659a8cd38ed2517201" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:35.176918 991 leader_election.cc:343] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:35.177273 989 leader_election.cc:343] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.177423 989 leader_election.cc:304] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.177704 2906 raft_consensus.cc:2749] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.178452 3141 raft_consensus.cc:515] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.181092 1203 tablet_bootstrap.cc:492] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.184149 1203 ts_tablet_manager.cc:1403] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.004s
I20251214 20:52:35.184988 1203 raft_consensus.cc:359] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.186910 1203 raft_consensus.cc:385] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.187105 1203 raft_consensus.cc:740] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.179414 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "140667afe42f42659a8cd38ed2517201" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.187727 1203 consensus_queue.cc:260] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.187990 2906 raft_consensus.cc:493] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.188419 3141 leader_election.cc:290] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.188424 2906 raft_consensus.cc:515] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.188467 768 raft_consensus.cc:3060] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.188968 2906 leader_election.cc:290] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.178491 1206 raft_consensus.cc:359] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.190083 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49778cfe80f6483389cfd421abee8ca7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.190027 1206 raft_consensus.cc:385] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.190773 1206 raft_consensus.cc:740] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.191298 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49778cfe80f6483389cfd421abee8ca7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:35.191516 989 leader_election.cc:343] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.192138 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e4dc66681ef48479ae3cb95b091564a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.191336 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b567effdb6834f7b87ce4aba9e555913 [id=c216b38d4fce484da23d096493ac2934]
W20251214 20:52:35.192938 991 leader_election.cc:343] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.193130 991 leader_election.cc:304] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.192625 1206 consensus_queue.cc:260] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.193470 2906 raft_consensus.cc:2749] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:35.193892 848 leader_election.cc:343] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.184831 1202 tablet_bootstrap.cc:492] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.186955 851 leader_election.cc:304] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.194478 1203 ts_tablet_manager.cc:1434] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20251214 20:52:35.194350 1202 ts_tablet_manager.cc:1403] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.001s
I20251214 20:52:35.195034 2889 raft_consensus.cc:2804] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.195267 2889 raft_consensus.cc:697] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.195667 1202 raft_consensus.cc:359] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.196148 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_62fb3a07f4a04029a6080d9d0e35d151 [id=c2886db5a22c43b5b371935e9f47187a]
I20251214 20:52:35.196622 1203 tablet_bootstrap.cc:492] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:35.196681 1206 ts_tablet_manager.cc:1434] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20251214 20:52:35.195750 2889 consensus_queue.cc:237] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.199214 1203 tablet_bootstrap.cc:654] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.199734 712 leader_election.cc:304] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.201104 1206 tablet_bootstrap.cc:492] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:35.204974 3111 consensus_queue.cc:1048] T db0ad1aa4193467db74cd707e7084b71 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.201880 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e4dc66681ef48479ae3cb95b091564a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.205591 1051 raft_consensus.cc:3060] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:35.208272 713 consensus_peers.cc:597] T 492770a7b8b74ba49315f974cc54eedb P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 492770a7b8b74ba49315f974cc54eedb. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.209144 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_9321256005b04155b2f5a451ce5065fe [id=c5a7a8665d6647bdb19ed6ef27907fee]
I20251214 20:52:35.210294 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_17954423e5674f0eb5a9b53bc2e75913 [id=c81e9d8bbee64045af7fce5cc8f64779]
I20251214 20:52:35.214308 1051 raft_consensus.cc:2468] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.215003 1202 raft_consensus.cc:385] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.215204 1202 raft_consensus.cc:740] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.215588 1202 consensus_queue.cc:260] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:35.218393 713 consensus_peers.cc:597] T eb29fd8c85344ed8aa8d880e2ed240bc P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb29fd8c85344ed8aa8d880e2ed240bc. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.219250 2991 raft_consensus.cc:2749] T 140667afe42f42659a8cd38ed2517201 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.221347 1202 ts_tablet_manager.cc:1434] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20251214 20:52:35.221908 1202 tablet_bootstrap.cc:492] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:35.230525 1202 tablet_bootstrap.cc:654] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.237854 2991 raft_consensus.cc:493] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.235023 851 leader_election.cc:304] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.238052 2991 raft_consensus.cc:515] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.239966 2906 raft_consensus.cc:493] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.240129 2906 raft_consensus.cc:515] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.240415 2991 leader_election.cc:290] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.240751 2906 leader_election.cc:290] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.241477 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "336cb9d03fd44f70930662ee3c8d987a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:35.242146 991 leader_election.cc:343] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.243373 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc73d590531c409e9f60670435dc8223" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.244110 908 raft_consensus.cc:2468] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:35.244694 712 leader_election.cc:304] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:35.244598 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "336cb9d03fd44f70930662ee3c8d987a" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.245080 2991 raft_consensus.cc:2804] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.244172 3141 raft_consensus.cc:2804] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.245220 2991 raft_consensus.cc:493] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.245322 2991 raft_consensus.cc:3060] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.246793 3141 raft_consensus.cc:697] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.247004 1206 tablet_bootstrap.cc:654] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.247185 3141 consensus_queue.cc:237] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.248139 2991 raft_consensus.cc:515] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.249588 2991 leader_election.cc:290] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:35.249969 713 leader_election.cc:343] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: bc73d590531c409e9f60670435dc8223
I20251214 20:52:35.250402 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc73d590531c409e9f60670435dc8223" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:35.250661 908 raft_consensus.cc:3060] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.250967 1202 tablet_bootstrap.cc:492] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.252570 1202 ts_tablet_manager.cc:1403] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.001s
I20251214 20:52:35.253593 1202 raft_consensus.cc:359] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.253191 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_d8b86c6bc911454fac28b6384cf9ff3c [id=c97bcc9ac37c4413bae618ee1019e069]
I20251214 20:52:35.260040 1202 raft_consensus.cc:385] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.261301 1202 raft_consensus.cc:740] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.261358 2991 consensus_queue.cc:1048] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.260486 908 raft_consensus.cc:2468] T bc73d590531c409e9f60670435dc8223 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
W20251214 20:52:35.262478 989 leader_election.cc:343] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.262598 989 leader_election.cc:304] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.262970 2906 raft_consensus.cc:2749] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.263254 712 leader_election.cc:304] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:35.263598 3098 raft_consensus.cc:2804] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.263724 3098 raft_consensus.cc:697] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:35.263967 3098 consensus_queue.cc:237] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:35.265084 713 leader_election.cc:343] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: bc73d590531c409e9f60670435dc8223
I20251214 20:52:35.271103 3111 consensus_queue.cc:1048] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:35.274587 848 consensus_peers.cc:597] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.285120 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ee3c57e398fd468fa1e4e7cc22f6747a [id=ce8357dfd1024446a69a3ee196ab4b3f]
I20251214 20:52:35.285662 1203 tablet_bootstrap.cc:492] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.285857 1203 ts_tablet_manager.cc:1403] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.089s user 0.000s sys 0.006s
I20251214 20:52:35.286697 1203 raft_consensus.cc:359] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.286891 1203 raft_consensus.cc:385] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.286942 1203 raft_consensus.cc:740] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.287447 1203 consensus_queue.cc:260] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.288020 1203 ts_tablet_manager.cc:1434] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251214 20:52:35.288352 1203 tablet_bootstrap.cc:492] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:35.290274 1203 tablet_bootstrap.cc:654] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.295181 3111 consensus_queue.cc:1048] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.297456 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5b1cf9007e0b4c3bae65a57ec193d8fa [id=cffdf6b3ab8b4d109c4552b0472fe6e9]
I20251214 20:52:35.299355 1206 tablet_bootstrap.cc:492] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:35.300233 1206 ts_tablet_manager.cc:1403] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.001s
W20251214 20:52:35.303354 848 consensus_peers.cc:597] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.306447 1202 consensus_queue.cc:260] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.308096 1202 ts_tablet_manager.cc:1434] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.055s user 0.001s sys 0.001s
I20251214 20:52:35.309414 1202 tablet_bootstrap.cc:492] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:35.312161 1202 tablet_bootstrap.cc:654] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.317303 2786 consensus_queue.cc:1048] T dfe50a6219134431b0a025317932609b P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.321674 2786 raft_consensus.cc:493] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.321846 2786 raft_consensus.cc:515] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.322510 2786 leader_election.cc:290] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.323107 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8db4364b5023438bb43538855007c95e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.323350 1051 raft_consensus.cc:2468] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.324405 1203 tablet_bootstrap.cc:492] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.324373 851 leader_election.cc:304] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:35.324666 2786 raft_consensus.cc:2804] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.324749 2786 raft_consensus.cc:493] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.324810 2786 raft_consensus.cc:3060] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.330157 2786 raft_consensus.cc:515] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.331879 2786 leader_election.cc:290] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.332500 3169 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6713a20a82b434f8a8bc122113e26ad)
I20251214 20:52:35.332727 3169 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c27224391721467a8f46e467a980b37a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42865 } } peers { permanent_uuid: "e6713a20a82b434f8a8bc122113e26ad" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34819 } } peers { permanent_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 35099 } }
I20251214 20:52:35.333513 3169 leader_election.cc:290] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c27224391721467a8f46e467a980b37a (127.0.104.62:42865), e6713a20a82b434f8a8bc122113e26ad (127.0.104.61:34819)
I20251214 20:52:35.335541 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8db4364b5023438bb43538855007c95e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.335568 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8db4364b5023438bb43538855007c95e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.336449 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_3024ac839d9441db89239acebb00bc12 [id=d3229230c8844d2fb42cd81149462a40]
W20251214 20:52:35.340117 848 consensus_peers.cc:597] T 292a28b12df246a8bc855d81c530746e P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:35.340663 848 leader_election.cc:343] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.340839 2777 raft_consensus.cc:493] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.340953 2777 raft_consensus.cc:515] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.341594 2777 leader_election.cc:290] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:35.342357 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5157ef5f1fd541efbbd2887bf4505254" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.342866 848 leader_election.cc:343] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:35.344347 713 consensus_peers.cc:597] T 6a000951f61948ac85fdd431f3d162c2 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a000951f61948ac85fdd431f3d162c2. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.345712 1206 raft_consensus.cc:359] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.346926 1206 raft_consensus.cc:385] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.347352 1206 raft_consensus.cc:740] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.348791 1206 consensus_queue.cc:260] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.356491 2884 consensus_queue.cc:1048] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.360399 1206 ts_tablet_manager.cc:1434] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.059s user 0.003s sys 0.000s
I20251214 20:52:35.327656 2906 raft_consensus.cc:493] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.362942 2906 raft_consensus.cc:515] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.363339 1203 ts_tablet_manager.cc:1403] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.006s
I20251214 20:52:35.363675 2906 leader_election.cc:290] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.364187 1203 raft_consensus.cc:359] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.364683 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57a6232564194cbba12182f3a29c2ebb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.365566 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_0fc25951e7cc4aebac50ad75316773f5 [id=d4196eff6f0c4965bfb9aa723570f6f3]
I20251214 20:52:35.365873 1206 tablet_bootstrap.cc:492] T d2a57409d4f54f17be792e6f3f17d7b7 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:35.366398 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57a6232564194cbba12182f3a29c2ebb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.342463 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5157ef5f1fd541efbbd2887bf4505254" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.372253 1051 raft_consensus.cc:2468] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.346823 3119 raft_consensus.cc:493] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.372682 3119 raft_consensus.cc:515] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.373378 3119 leader_election.cc:290] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:35.365861 989 leader_election.cc:343] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.373742 851 leader_election.cc:304] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.374040 2786 raft_consensus.cc:2804] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251214 20:52:35.374127 991 leader_election.cc:343] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.373387 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_14f3cdeac59a4c20988d0e23744d2fbd [id=d53e0d2c6a7b4dc89f93ddd315732229]
I20251214 20:52:35.374366 991 leader_election.cc:304] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.374385 2786 raft_consensus.cc:493] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.374475 2786 raft_consensus.cc:3060] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.375108 3119 raft_consensus.cc:2749] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.375643 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e5fff1fc84419ab3d3131cb3a12d92" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:35.376245 991 leader_election.cc:343] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.376595 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e5fff1fc84419ab3d3131cb3a12d92" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.377202 989 leader_election.cc:343] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.377301 989 leader_election.cc:304] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.377753 3119 raft_consensus.cc:2749] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.333827 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8db4364b5023438bb43538855007c95e" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.377905 2786 raft_consensus.cc:515] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.378093 1050 raft_consensus.cc:3060] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.377868 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_26fdf6b490ab4f6cb488f406406a190c [id=d64c201020c34baf8cf82672f00aa29e]
I20251214 20:52:35.379737 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5157ef5f1fd541efbbd2887bf4505254" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.379526 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5157ef5f1fd541efbbd2887bf4505254" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.379921 1051 raft_consensus.cc:3060] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:35.380600 848 leader_election.cc:343] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.380954 2786 leader_election.cc:290] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:35.382135 1050 raft_consensus.cc:2468] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.382334 3039 raft_consensus.cc:493] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.382475 3039 raft_consensus.cc:515] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.382567 1203 raft_consensus.cc:385] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.382628 1203 raft_consensus.cc:740] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.382848 1203 consensus_queue.cc:260] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.383237 3039 leader_election.cc:290] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.385207 851 leader_election.cc:304] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
W20251214 20:52:35.386011 713 leader_election.cc:343] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 088c72a6f5014b6fbc6ee060b6b7a590
I20251214 20:52:35.382210 1051 raft_consensus.cc:2468] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.386447 851 leader_election.cc:304] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.386911 2777 raft_consensus.cc:2804] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.387051 2777 raft_consensus.cc:697] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.387162 3108 raft_consensus.cc:2804] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.387310 3108 raft_consensus.cc:697] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.387333 2777 consensus_queue.cc:237] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.387588 3108 consensus_queue.cc:237] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.390033 1203 ts_tablet_manager.cc:1434] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.027s user 0.000s sys 0.002s
I20251214 20:52:35.390865 1203 tablet_bootstrap.cc:492] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:35.391297 3111 raft_consensus.cc:493] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.391252 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "088c72a6f5014b6fbc6ee060b6b7a590" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.391434 3111 raft_consensus.cc:515] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.392000 3111 leader_election.cc:290] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.392494 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "054cfd587d19409fbdf7f4848bcaf3f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.392566 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "054cfd587d19409fbdf7f4848bcaf3f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.392731 1051 raft_consensus.cc:2468] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:35.393076 848 leader_election.cc:343] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.393368 851 leader_election.cc:304] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.393815 3111 raft_consensus.cc:2804] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.393889 3111 raft_consensus.cc:493] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.393926 1203 tablet_bootstrap.cc:654] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.393944 3111 raft_consensus.cc:3060] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.397073 3111 raft_consensus.cc:515] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.397810 3111 leader_election.cc:290] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.398748 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "054cfd587d19409fbdf7f4848bcaf3f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.399041 1051 raft_consensus.cc:3060] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.402444 1051 raft_consensus.cc:2468] T 054cfd587d19409fbdf7f4848bcaf3f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.391487 908 raft_consensus.cc:2468] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:35.403017 851 leader_election.cc:304] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:35.403381 3141 raft_consensus.cc:2804] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.403719 3141 raft_consensus.cc:697] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.404183 3141 consensus_queue.cc:237] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.405197 712 leader_election.cc:304] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:35.405534 3098 raft_consensus.cc:2804] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.405716 3098 raft_consensus.cc:493] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.405889 3098 raft_consensus.cc:3060] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.407583 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "054cfd587d19409fbdf7f4848bcaf3f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:35.408210 848 leader_election.cc:343] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.409507 3098 raft_consensus.cc:515] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.410835 1206 tablet_bootstrap.cc:654] T d2a57409d4f54f17be792e6f3f17d7b7 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:35.416172 713 leader_election.cc:343] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 088c72a6f5014b6fbc6ee060b6b7a590
I20251214 20:52:35.417065 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_bbd8d7f1a4b04eeeaf8ccd7e060b7476 [id=d6da680c34f440b2b0d1182a3f8ac19d]
I20251214 20:52:35.418588 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "088c72a6f5014b6fbc6ee060b6b7a590" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:35.418895 908 raft_consensus.cc:3060] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.418681 3098 leader_election.cc:290] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.421990 3098 raft_consensus.cc:493] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:35.425087 1287 leader_election.cc:336] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer c27224391721467a8f46e467a980b37a (127.0.104.62:42865): Network error: Client connection negotiation failed: client connection to 127.0.104.62:42865: connect: Connection refused (error 111)
I20251214 20:52:35.423630 3098 raft_consensus.cc:515] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:35.426034 713 consensus_peers.cc:597] T b1f4ea72c6274569a98cab6afdc55d55 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1f4ea72c6274569a98cab6afdc55d55. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.427623 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_adb418bf04534b868c044988fb97e906 [id=d7e403a944e94d0d8c9d6ba115723ea4]
I20251214 20:52:35.431942 3098 leader_election.cc:290] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:35.432988 712 leader_election.cc:343] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: e9f6da5d29294f08a86c96c6821dda10
I20251214 20:52:35.433401 3119 raft_consensus.cc:493] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.433563 3119 raft_consensus.cc:515] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.434255 3119 leader_election.cc:290] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.434821 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41effce1ac504f27b619a4d9950202f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.434926 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41effce1ac504f27b619a4d9950202f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.435235 3108 raft_consensus.cc:493] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:35.435411 989 leader_election.cc:343] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.435456 3108 raft_consensus.cc:515] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:35.436118 713 leader_election.cc:343] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: e9f6da5d29294f08a86c96c6821dda10
I20251214 20:52:35.436228 713 leader_election.cc:304] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:35.436715 991 leader_election.cc:343] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.436911 991 leader_election.cc:304] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.438196 1202 tablet_bootstrap.cc:492] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.438179 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d171301941fa43908e05984c3b016007" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.438468 3039 raft_consensus.cc:2749] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:35.438666 848 leader_election.cc:343] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.438745 3119 raft_consensus.cc:2749] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.438386 1202 ts_tablet_manager.cc:1403] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.130s user 0.015s sys 0.011s
I20251214 20:52:35.439296 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_e4ed29947e6544b9a23370d584496d4a [id=daf3f737290e4b789b9f81ed0f0fa375]
I20251214 20:52:35.440057 1202 raft_consensus.cc:359] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.440274 1202 raft_consensus.cc:385] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.440346 1202 raft_consensus.cc:740] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.440555 1202 consensus_queue.cc:260] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.441033 1202 ts_tablet_manager.cc:1434] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:35.441576 1202 tablet_bootstrap.cc:492] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:35.441295 3108 leader_election.cc:290] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.442909 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d171301941fa43908e05984c3b016007" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.443161 1051 raft_consensus.cc:2468] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.444088 851 leader_election.cc:304] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.444362 3108 raft_consensus.cc:2804] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.444444 3108 raft_consensus.cc:493] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.444499 3108 raft_consensus.cc:3060] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.445545 1202 tablet_bootstrap.cc:654] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.446925 3108 raft_consensus.cc:515] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.447750 3108 leader_election.cc:290] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.448629 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d171301941fa43908e05984c3b016007" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.449151 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d171301941fa43908e05984c3b016007" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
W20251214 20:52:35.449270 848 leader_election.cc:343] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.449374 1051 raft_consensus.cc:3060] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.452097 908 raft_consensus.cc:2468] T 088c72a6f5014b6fbc6ee060b6b7a590 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:35.452445 1051 raft_consensus.cc:2468] T d171301941fa43908e05984c3b016007 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.452929 851 leader_election.cc:304] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.453771 2786 raft_consensus.cc:2804] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.453922 2786 raft_consensus.cc:697] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.455207 712 leader_election.cc:304] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:35.454841 2786 consensus_queue.cc:237] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.456254 2991 raft_consensus.cc:2804] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.456403 2991 raft_consensus.cc:697] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:35.456641 2991 consensus_queue.cc:237] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.459355 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_5c7e3a65102c45a1a23d29b95427fae5 [id=db0d9e33bf054daabddadd1248243609]
I20251214 20:52:35.463306 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589]
I20251214 20:52:35.464722 1206 tablet_bootstrap.cc:492] T d2a57409d4f54f17be792e6f3f17d7b7 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:35.465061 1206 ts_tablet_manager.cc:1403] T d2a57409d4f54f17be792e6f3f17d7b7 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.100s user 0.005s sys 0.004s
I20251214 20:52:35.466509 1206 raft_consensus.cc:359] T d2a57409d4f54f17be792e6f3f17d7b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.466899 1206 raft_consensus.cc:385] T d2a57409d4f54f17be792e6f3f17d7b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.467042 1206 raft_consensus.cc:740] T d2a57409d4f54f17be792e6f3f17d7b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.467516 1206 consensus_queue.cc:260] T d2a57409d4f54f17be792e6f3f17d7b7 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.468133 1203 tablet_bootstrap.cc:492] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.468295 1203 ts_tablet_manager.cc:1403] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.000s
I20251214 20:52:35.469043 1203 raft_consensus.cc:359] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.469221 1203 raft_consensus.cc:385] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.469280 1203 raft_consensus.cc:740] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.469486 1203 consensus_queue.cc:260] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.469973 1203 ts_tablet_manager.cc:1434] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:35.470319 1203 tablet_bootstrap.cc:492] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:35.472505 1203 tablet_bootstrap.cc:654] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.476109 1202 tablet_bootstrap.cc:492] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.476272 1202 ts_tablet_manager.cc:1403] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20251214 20:52:35.476979 1202 raft_consensus.cc:359] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.477142 1202 raft_consensus.cc:385] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.477200 1202 raft_consensus.cc:740] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.477376 1202 consensus_queue.cc:260] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.477856 1202 ts_tablet_manager.cc:1434] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:35.478130 1202 tablet_bootstrap.cc:492] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:35.485649 1206 ts_tablet_manager.cc:1434] T d2a57409d4f54f17be792e6f3f17d7b7 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20251214 20:52:35.490329 1206 tablet_bootstrap.cc:492] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:35.490720 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_d80c747f5a704a608b5e28b5e5d247b1 [id=e09da957fd5b4879a87ffb8c1cdd4bbb]
I20251214 20:52:35.491518 2991 raft_consensus.cc:493] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2e8fd1cb731446d88f8e510c7a52999)
I20251214 20:52:35.491752 2991 raft_consensus.cc:515] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.492460 2991 leader_election.cc:290] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:35.493474 713 leader_election.cc:343] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 8d9f74fb2c6244a6887e3c922262bbab
I20251214 20:52:35.493949 1202 tablet_bootstrap.cc:654] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:35.494881 712 leader_election.cc:343] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 8d9f74fb2c6244a6887e3c922262bbab
I20251214 20:52:35.494974 712 leader_election.cc:304] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.495359 2991 raft_consensus.cc:2749] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:35.495275 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_88881a27865d4b0aa6db6a2b736f0c17 [id=e0d1675617c74aa8a8afd81bd90f9639]
I20251214 20:52:35.501698 1203 tablet_bootstrap.cc:492] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.501929 1203 ts_tablet_manager.cc:1403] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
I20251214 20:52:35.502794 1203 raft_consensus.cc:359] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.503029 1203 raft_consensus.cc:385] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.503114 1203 raft_consensus.cc:740] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.503410 1203 consensus_queue.cc:260] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.503938 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_27579b7155b140b3bdda5014c1eb57a8 [id=e24a6c9e5c9049edba5e42e22d22a831]
I20251214 20:52:35.507639 1203 ts_tablet_manager.cc:1434] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251214 20:52:35.508019 1203 tablet_bootstrap.cc:492] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:35.509866 1203 tablet_bootstrap.cc:654] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:35.518282 848 consensus_peers.cc:597] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.520493 1206 tablet_bootstrap.cc:654] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.522773 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_1aadd2d8580e419fb5a3767b09353c25 [id=e4d01eb0c006487bb2f14fb3e14dd18f]
I20251214 20:52:35.528927 1202 tablet_bootstrap.cc:492] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.529348 1202 ts_tablet_manager.cc:1403] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.002s
I20251214 20:52:35.530419 1202 raft_consensus.cc:359] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.530692 1202 raft_consensus.cc:385] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.530788 1202 raft_consensus.cc:740] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.531082 1202 consensus_queue.cc:260] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.531781 1202 ts_tablet_manager.cc:1434] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:35.532368 1202 tablet_bootstrap.cc:492] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:35.536484 1202 tablet_bootstrap.cc:654] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.540691 3119 raft_consensus.cc:493] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.540901 3119 raft_consensus.cc:515] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.541744 3119 leader_election.cc:290] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.542600 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f22ac24c38e4c64bb26c54ae4d87dfb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.542968 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f22ac24c38e4c64bb26c54ae4d87dfb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:35.543721 989 leader_election.cc:343] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:35.544134 991 leader_election.cc:343] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.544243 991 leader_election.cc:304] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.544689 3119 raft_consensus.cc:2749] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.544932 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b3dbc2e42cc64d9d96c48ecc1b474480 [id=e78b07fcb7a341fdbd0c244409c3c445]
I20251214 20:52:35.545718 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_cb922bf98617480aa9363c3f08d918f3 [id=e815640c7ece4361800a22bc749915e9]
I20251214 20:52:35.546335 1203 tablet_bootstrap.cc:492] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.546518 1203 ts_tablet_manager.cc:1403] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.005s
I20251214 20:52:35.547312 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_b58a6dfd93fb438684f7564325241f3e [id=e85a8088178744e0a19699f5d680b4fa]
I20251214 20:52:35.547799 1203 raft_consensus.cc:359] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.547986 1203 raft_consensus.cc:385] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.548041 1203 raft_consensus.cc:740] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.549141 1206 tablet_bootstrap.cc:492] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:35.549428 1206 ts_tablet_manager.cc:1403] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.000s
I20251214 20:52:35.550938 1206 raft_consensus.cc:359] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.548269 1203 consensus_queue.cc:260] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.551402 1206 raft_consensus.cc:385] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.551553 1206 raft_consensus.cc:740] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.554009 1206 consensus_queue.cc:260] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.555706 1203 ts_tablet_manager.cc:1434] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:35.556049 1203 tablet_bootstrap.cc:492] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:35.557875 3119 raft_consensus.cc:493] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.558008 3119 raft_consensus.cc:515] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.558188 2991 raft_consensus.cc:493] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.558336 2991 raft_consensus.cc:515] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.558477 1203 tablet_bootstrap.cc:654] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.558789 3119 leader_election.cc:290] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.558892 2991 leader_election.cc:290] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.559724 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c2a6c6c0b6e46bcb7658fbeced6f738" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.560346 1202 tablet_bootstrap.cc:492] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.560510 1202 ts_tablet_manager.cc:1403] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
W20251214 20:52:35.560587 991 leader_election.cc:343] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.561542 1202 raft_consensus.cc:359] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.562749 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c2a6c6c0b6e46bcb7658fbeced6f738" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.562533 3119 raft_consensus.cc:493] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.563272 3119 raft_consensus.cc:515] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.563920 3119 leader_election.cc:290] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.564060 1202 raft_consensus.cc:385] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251214 20:52:35.564105 712 leader_election.cc:343] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 47e813373fbd482298312ead4f037c01
I20251214 20:52:35.564164 1202 raft_consensus.cc:740] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
W20251214 20:52:35.564602 713 leader_election.cc:343] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 47e813373fbd482298312ead4f037c01
I20251214 20:52:35.564491 1202 consensus_queue.cc:260] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.564708 713 leader_election.cc:304] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.564946 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34df3a34b15b4fe19bd12f785b07a033" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.565447 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34df3a34b15b4fe19bd12f785b07a033" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:35.566004 991 leader_election.cc:343] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.567735 2991 raft_consensus.cc:2749] T 47e813373fbd482298312ead4f037c01 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:35.568403 989 leader_election.cc:343] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.568528 989 leader_election.cc:304] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:35.568975 989 leader_election.cc:343] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.569067 989 leader_election.cc:304] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.569403 2906 raft_consensus.cc:2749] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.569558 2906 raft_consensus.cc:2749] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.572829 1202 ts_tablet_manager.cc:1434] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20251214 20:52:35.575559 1206 ts_tablet_manager.cc:1434] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20251214 20:52:35.578538 1206 tablet_bootstrap.cc:492] T 9f04336d2c84439b9003df19d3541273 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:35.582373 3119 raft_consensus.cc:493] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.582655 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_e277c40da9e24b7fabb8edd25ec3413d [id=e973536efad243e5b94f5994d57325b4]
I20251214 20:52:35.583315 3119 raft_consensus.cc:515] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.586768 3119 leader_election.cc:290] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.587106 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9488c3505f5443998531cf749722be9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.587925 1202 tablet_bootstrap.cc:492] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:35.588055 991 leader_election.cc:343] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.590466 1202 tablet_bootstrap.cc:654] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.593369 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9488c3505f5443998531cf749722be9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.600250 2777 raft_consensus.cc:493] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.600412 2777 raft_consensus.cc:515] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.600982 2777 leader_election.cc:290] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.601557 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825baba561ee4a8ebd44f75d8c51706a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.602185 848 leader_election.cc:343] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.602181 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825baba561ee4a8ebd44f75d8c51706a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.602363 1051 raft_consensus.cc:2468] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.602716 851 leader_election.cc:304] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.603072 2777 raft_consensus.cc:2804] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.603166 2777 raft_consensus.cc:493] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.603232 2777 raft_consensus.cc:3060] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.603613 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_2fca6840abf741ed94baef488b3640dc [id=eb416f50656b403e98245036103dcdba]
I20251214 20:52:35.605757 2777 raft_consensus.cc:515] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.606323 2777 leader_election.cc:290] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:35.606665 713 consensus_peers.cc:597] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b47ca1a03944cf1847d3578856dafa7. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:35.606979 989 leader_election.cc:343] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.607100 989 leader_election.cc:304] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.607229 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825baba561ee4a8ebd44f75d8c51706a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.607434 1052 raft_consensus.cc:3060] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.607774 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "825baba561ee4a8ebd44f75d8c51706a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.607842 3119 raft_consensus.cc:2749] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:35.608338 848 leader_election.cc:343] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.609824 3108 consensus_queue.cc:1048] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.610705 1052 raft_consensus.cc:2468] T 825baba561ee4a8ebd44f75d8c51706a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.611483 851 leader_election.cc:304] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.612030 2777 raft_consensus.cc:2804] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.611801 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_8f5d31355346492989caa148a8f1440b [id=ec2fd20884ff4bdca7be9d32d3bfe9ae]
I20251214 20:52:35.612552 2777 raft_consensus.cc:697] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.612864 2777 consensus_queue.cc:237] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.613291 1202 tablet_bootstrap.cc:492] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.613476 1202 ts_tablet_manager.cc:1403] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.013s
W20251214 20:52:35.615777 713 consensus_peers.cc:597] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc7353593bda4fceaa389c393ddc19c6. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.615689 1202 raft_consensus.cc:359] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.615883 1202 raft_consensus.cc:385] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.615978 1202 raft_consensus.cc:740] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.616202 1202 consensus_queue.cc:260] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.617592 1203 tablet_bootstrap.cc:492] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.617888 1203 ts_tablet_manager.cc:1403] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.003s
I20251214 20:52:35.618685 1206 tablet_bootstrap.cc:654] T 9f04336d2c84439b9003df19d3541273 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.619401 1203 raft_consensus.cc:359] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.619889 1203 raft_consensus.cc:385] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.620033 1203 raft_consensus.cc:740] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.615118 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_f86bf75508ca43ada4372af83098ae35 [id=ecd6b2f07b474967adf0662a8b89979b]
I20251214 20:52:35.620491 1203 consensus_queue.cc:260] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.622117 1202 ts_tablet_manager.cc:1434] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251214 20:52:35.622963 1202 tablet_bootstrap.cc:492] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:35.624780 1203 ts_tablet_manager.cc:1434] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251214 20:52:35.625582 1202 tablet_bootstrap.cc:654] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.627211 1203 tablet_bootstrap.cc:492] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:35.627424 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_fe0d773656e24c0e9d6e2e36caeaa9f8 [id=ed335fb680bc441184e1682200ee68be]
I20251214 20:52:35.633898 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_691250c9129d453585ad50b32a1e340e [id=edcea358cf1c4046a473b27b94ba56ac]
I20251214 20:52:35.638584 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_01b1f692968d4dbe8d92746b6017a8b3 [id=eefab419a1984e92997ff7429a6dd368]
I20251214 20:52:35.647584 1203 tablet_bootstrap.cc:654] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.651742 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_3c676f861c28408f99faac2ab8f7eec1 [id=ef22180b36a241fea13d0e622c02b50a]
W20251214 20:52:35.659643 848 consensus_peers.cc:597] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.659904 3108 consensus_queue.cc:1048] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.658329 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3 [id=f1338e81f4ec4dce87130f2ddbe98041]
I20251214 20:52:35.663370 3108 raft_consensus.cc:493] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.663548 3108 raft_consensus.cc:515] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.664222 3108 leader_election.cc:290] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:35.664795 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28d34670fdce4242a3feccf830a12880" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.665377 848 leader_election.cc:343] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.665992 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28d34670fdce4242a3feccf830a12880" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.666095 1206 tablet_bootstrap.cc:492] T 9f04336d2c84439b9003df19d3541273 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:35.666201 1050 raft_consensus.cc:2468] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.666354 1206 ts_tablet_manager.cc:1403] T 9f04336d2c84439b9003df19d3541273 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.010s
I20251214 20:52:35.666508 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8]
I20251214 20:52:35.666774 851 leader_election.cc:304] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.667430 2777 raft_consensus.cc:2804] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.667641 2777 raft_consensus.cc:493] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.667876 2777 raft_consensus.cc:3060] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.667814 1206 raft_consensus.cc:359] T 9f04336d2c84439b9003df19d3541273 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.668181 1206 raft_consensus.cc:385] T 9f04336d2c84439b9003df19d3541273 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.668401 1206 raft_consensus.cc:740] T 9f04336d2c84439b9003df19d3541273 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.668910 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_a426957d72cc493393637b00a3b1e12d [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde]
I20251214 20:52:35.669257 1206 consensus_queue.cc:260] T 9f04336d2c84439b9003df19d3541273 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.672770 2777 raft_consensus.cc:515] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.673437 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_9151bdce51684c39901786aef1c63ec5 [id=f4157d7b9fc848a8b7fa08742b456167]
I20251214 20:52:35.674222 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980]
I20251214 20:52:35.677714 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28d34670fdce4242a3feccf830a12880" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.677946 1051 raft_consensus.cc:3060] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.678105 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_81e31e2b08a1484c8adb4f139e0edfd3 [id=f6a2470a04cf40e0b3d04ec337fe328b]
I20251214 20:52:35.678802 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]
W20251214 20:52:35.679605 848 consensus_peers.cc:597] T d2688515bd1441fbad7516f8343ca775 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.684386 1206 ts_tablet_manager.cc:1434] T 9f04336d2c84439b9003df19d3541273 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251214 20:52:35.684871 1202 tablet_bootstrap.cc:492] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.685103 1202 ts_tablet_manager.cc:1403] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.010s
I20251214 20:52:35.685618 775 raft_consensus.cc:3060] T 9f04336d2c84439b9003df19d3541273 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.690563 1202 raft_consensus.cc:359] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.691905 1202 raft_consensus.cc:385] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.691820 3098 consensus_queue.cc:1048] T 3b47ca1a03944cf1847d3578856dafa7 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.692205 1202 raft_consensus.cc:740] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.692754 1202 consensus_queue.cc:260] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.694339 1202 ts_tablet_manager.cc:1434] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251214 20:52:35.695215 3119 raft_consensus.cc:493] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.695365 3119 raft_consensus.cc:515] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.695701 1203 tablet_bootstrap.cc:492] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.695832 1203 ts_tablet_manager.cc:1403] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.069s user 0.007s sys 0.000s
I20251214 20:52:35.696425 1202 tablet_bootstrap.cc:492] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:35.696592 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43fe3bb1ed3941ce9078376f8f478408" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.697196 989 leader_election.cc:343] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.698817 3098 raft_consensus.cc:493] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2e8fd1cb731446d88f8e510c7a52999)
I20251214 20:52:35.698979 3098 raft_consensus.cc:515] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.699594 3098 leader_election.cc:290] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:35.700795 712 leader_election.cc:343] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 1dca369dc5f54a4ea1783e334c5670d6
I20251214 20:52:35.701737 1202 tablet_bootstrap.cc:654] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.704547 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_e2a035220b924122a538db32c77a683b [id=f97b0be6362b47b49c15807bdf090e66]
W20251214 20:52:35.704694 848 consensus_peers.cc:597] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.705305 3119 leader_election.cc:290] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.706883 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43fe3bb1ed3941ce9078376f8f478408" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:35.706966 713 leader_election.cc:343] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 1dca369dc5f54a4ea1783e334c5670d6
I20251214 20:52:35.707103 713 leader_election.cc:304] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:35.707471 991 leader_election.cc:343] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.707583 991 leader_election.cc:304] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.707610 2991 raft_consensus.cc:2749] T 1dca369dc5f54a4ea1783e334c5670d6 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:35.707932 3119 raft_consensus.cc:2749] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.708256 2786 consensus_queue.cc:1048] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.708531 1203 raft_consensus.cc:359] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.708707 1203 raft_consensus.cc:385] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.708755 1203 raft_consensus.cc:740] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.708938 1203 consensus_queue.cc:260] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.709622 1203 ts_tablet_manager.cc:1434] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251214 20:52:35.716853 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_8d0c148da21f4137b09c9631e0b3f9bf [id=fa40006b630d491e88479cd9f61def2e]
I20251214 20:52:35.720064 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_403e5819717949e8bffa1ed844a68964 [id=faaedb3e871c491691ecec42c3ab9d8f]
I20251214 20:52:35.722018 1051 raft_consensus.cc:2468] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.722366 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_e95c404ef11b4cf79b5a81ec1eddb57a [id=fb52e77ee3ec48c39afd588c8de11764]
I20251214 20:52:35.723646 851 leader_election.cc:304] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:35.724102 3108 raft_consensus.cc:2804] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.724411 3108 raft_consensus.cc:697] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.724989 3108 consensus_queue.cc:237] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.725606 1206 tablet_bootstrap.cc:492] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:35.729866 848 consensus_peers.cc:597] T 3915abc53b084854912a8e4119eefc7e P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.734400 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778]
I20251214 20:52:35.736078 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]
I20251214 20:52:35.736393 3119 raft_consensus.cc:493] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.736584 3119 raft_consensus.cc:515] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.737167 3119 leader_election.cc:290] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.737731 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25180d5352204cdca3c49f907a587cb2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.738432 989 leader_election.cc:343] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.739535 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25180d5352204cdca3c49f907a587cb2" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.740764 1206 tablet_bootstrap.cc:654] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.742652 1203 tablet_bootstrap.cc:492] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:35.743790 991 leader_election.cc:343] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.743912 991 leader_election.cc:304] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.744192 3119 raft_consensus.cc:2749] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.759768 2946 catalog_manager.cc:679] Loaded metadata for table default.Table_1191163db85341c5b41c4baf61bc8df5 [id=fcb196b8ebb14e1785ea355e7d406b8a]
W20251214 20:52:35.761920 848 consensus_peers.cc:597] T 1d7aa508b11944aa8e268741c1998a9e P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.762961 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28d34670fdce4242a3feccf830a12880" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.764662 2777 leader_election.cc:290] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:35.765842 848 leader_election.cc:343] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.766829 1203 tablet_bootstrap.cc:654] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:35.772249 713 consensus_peers.cc:597] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe24e4f4e74040249068505058c15622. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.775195 3141 consensus_queue.cc:1048] T f7f06eded6c340c58101fcc644ebcf50 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.777062 1202 tablet_bootstrap.cc:492] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.777704 1202 ts_tablet_manager.cc:1403] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.081s user 0.001s sys 0.008s
I20251214 20:52:35.779290 1202 raft_consensus.cc:359] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.779726 1202 raft_consensus.cc:385] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.779883 1202 raft_consensus.cc:740] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.780371 1202 consensus_queue.cc:260] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.781605 1202 ts_tablet_manager.cc:1434] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251214 20:52:35.782249 1202 tablet_bootstrap.cc:492] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:35.785022 3098 consensus_queue.cc:1048] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.785274 1202 tablet_bootstrap.cc:654] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.788503 1206 tablet_bootstrap.cc:492] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:35.788777 1206 ts_tablet_manager.cc:1403] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.008s
I20251214 20:52:35.790033 1206 raft_consensus.cc:359] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.790310 1206 raft_consensus.cc:385] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.790421 1206 raft_consensus.cc:740] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.792064 1206 consensus_queue.cc:260] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.797796 1206 ts_tablet_manager.cc:1434] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251214 20:52:35.801277 1206 tablet_bootstrap.cc:492] T 292a28b12df246a8bc855d81c530746e P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:35.808557 3119 raft_consensus.cc:493] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.808756 3119 raft_consensus.cc:515] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.809419 3119 leader_election.cc:290] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.810177 1206 tablet_bootstrap.cc:654] T 292a28b12df246a8bc855d81c530746e P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.810348 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b482043af42485eb4c73d6eb864a963" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:35.810995 991 leader_election.cc:343] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.811501 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b482043af42485eb4c73d6eb864a963" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.812065 989 leader_election.cc:343] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.812161 989 leader_election.cc:304] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.812815 2906 raft_consensus.cc:2749] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.822808 1202 tablet_bootstrap.cc:492] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:35.823103 1202 ts_tablet_manager.cc:1403] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.004s
I20251214 20:52:35.823830 2946 tablet_loader.cc:96] loaded metadata for tablet 003bacf23a204e048509e0ceb6b80b9c (table default.Table_78f00e8dd81a485cae8269bac0d08866 [id=bc52dc01429e446b9726bc62fd9b24b3])
I20251214 20:52:35.824470 2946 tablet_loader.cc:96] loaded metadata for tablet 009081f974344dac83e66cdf2294ceac (table default.Table_2fca6840abf741ed94baef488b3640dc [id=eb416f50656b403e98245036103dcdba])
I20251214 20:52:35.824460 1202 raft_consensus.cc:359] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.824813 1202 raft_consensus.cc:385] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.824916 2946 tablet_loader.cc:96] loaded metadata for tablet 00cf456eda464f5ba22d6893f73b5cba (table default.Table_6838d68b72c042a293fd6c13096e6b78 [id=065328c700574078a1380f42b9848e2c])
I20251214 20:52:35.824991 1202 raft_consensus.cc:740] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.825637 2946 tablet_loader.cc:96] loaded metadata for tablet 016581519f4c4e0d832b49346d55b41c (table default.Table_531eb871a88242c5b91423a2c6abc3b6 [id=9afdb569356648bb84089b1738dfc359])
I20251214 20:52:35.825502 1202 consensus_queue.cc:260] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.826324 2946 tablet_loader.cc:96] loaded metadata for tablet 018452d555e04d1082da2cec1812a1f5 (table default.Table_624b9386a78b44e994cde78539ccedc4 [id=7ea13a9516b340dabbb4916b419b8c03])
I20251214 20:52:35.827322 1202 ts_tablet_manager.cc:1434] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251214 20:52:35.829243 1202 tablet_bootstrap.cc:492] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:35.829559 2946 tablet_loader.cc:96] loaded metadata for tablet 03ab5235046c4ec8afc4bc6730d6cfa8 (table default.Table_cca1629ea2974eddb3a2fab0ae9f4500 [id=7ceb70a9b9b0458692720cd351d9a3c6])
I20251214 20:52:35.832366 2946 tablet_loader.cc:96] loaded metadata for tablet 055b1cc69c9847ee81ee106fead48fa5 (table default.Table_bbd8d7f1a4b04eeeaf8ccd7e060b7476 [id=d6da680c34f440b2b0d1182a3f8ac19d])
I20251214 20:52:35.832792 2946 tablet_loader.cc:96] loaded metadata for tablet 0634838779854905b64dcdd028deb1e1 (table default.Table_5a79c363f13a4849a6be94ea935e5b89 [id=4093d9a107134e4b90970ddecc931769])
I20251214 20:52:35.833439 2946 tablet_loader.cc:96] loaded metadata for tablet 06ebe9f717bf43108748605cf4f961b7 (table default.Table_ae43058f9a39467b9919a9e1c335602f [id=7c91475975864cac93bfeb4d13173f1b])
I20251214 20:52:35.836320 2946 tablet_loader.cc:96] loaded metadata for tablet 078dff96c9704e36b6b26692f51b890e (table default.Table_176fae22b3bf4a159ac799ec6339ea16 [id=08bd7456e0db4786a4f2508d2709abea])
I20251214 20:52:35.836802 2946 tablet_loader.cc:96] loaded metadata for tablet 07a8662a634a4befbd2d6251000d2059 (table default.Table_76f578444d71410a93c3f78ff50b7951 [id=6157817e96d642d09f9941ad7f7e82ee])
I20251214 20:52:35.837348 2946 tablet_loader.cc:96] loaded metadata for tablet 07e5cf36a7cc476fafd3473a1de9eced (table default.Table_40e2218a9f58420e9a09ccfbda122485 [id=9fa32d9e7b7c49baa67e36e0eb996e2e])
I20251214 20:52:35.837772 2946 tablet_loader.cc:96] loaded metadata for tablet 082ec70b8ee646b898d9201ad073d292 (table default.Table_e3feb6d6cacb48dca8455eacf4cfb84b [id=8f720514e13b45689817a9811fb624d9])
I20251214 20:52:35.840132 2946 tablet_loader.cc:96] loaded metadata for tablet 098dda4fe9d94fd8be2380d3f840cc92 (table default.Table_ac56c984d6f5476cb187409e4e7575b6 [id=1350143e22c44ac59f5ac65a3c2f5763])
I20251214 20:52:35.842224 2946 tablet_loader.cc:96] loaded metadata for tablet 0c62ffb71fff43d4812eb86f40ce1c58 (table default.Table_cdf0b37254924fa18a712b2bd472fb46 [id=3322c0d4eb1a409da134030cd0a993fe])
W20251214 20:52:35.850307 713 consensus_peers.cc:597] T a0976ffcc649461fac74e20b5e402275 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0976ffcc649461fac74e20b5e402275. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.858142 2946 tablet_loader.cc:96] loaded metadata for tablet 0d0f7bd831b4474a90288ac7fa600436 (table default.Table_d8b86c6bc911454fac28b6384cf9ff3c [id=c97bcc9ac37c4413bae618ee1019e069])
I20251214 20:52:35.858597 775 raft_consensus.cc:3060] T d2a57409d4f54f17be792e6f3f17d7b7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.858960 3141 consensus_queue.cc:1048] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.863538 1206 tablet_bootstrap.cc:492] T 292a28b12df246a8bc855d81c530746e P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:35.863977 1206 ts_tablet_manager.cc:1403] T 292a28b12df246a8bc855d81c530746e P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.001s
W20251214 20:52:35.865116 848 consensus_peers.cc:597] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.865372 1206 raft_consensus.cc:359] T 292a28b12df246a8bc855d81c530746e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.865808 1206 raft_consensus.cc:385] T 292a28b12df246a8bc855d81c530746e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.865811 2946 tablet_loader.cc:96] loaded metadata for tablet 0dbfe8ee68bf4a2c88489b1f81b36991 (table default.Table_53bf327c948f4cebbe6096266ac301f8 [id=6a279e57008549c593df838211cc9cc1])
I20251214 20:52:35.865947 1206 raft_consensus.cc:740] T 292a28b12df246a8bc855d81c530746e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.866343 1206 consensus_queue.cc:260] T 292a28b12df246a8bc855d81c530746e P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.868834 1206 ts_tablet_manager.cc:1434] T 292a28b12df246a8bc855d81c530746e P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251214 20:52:35.869328 2777 raft_consensus.cc:493] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.869503 2777 raft_consensus.cc:515] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.870095 2777 leader_election.cc:290] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.870539 1202 tablet_bootstrap.cc:654] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.870879 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bed2c326906486e99300e05033c9c0f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.871074 1050 raft_consensus.cc:2468] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.871703 851 leader_election.cc:304] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:35.871845 1206 tablet_bootstrap.cc:492] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:35.872087 2889 raft_consensus.cc:2804] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.872239 2884 raft_consensus.cc:493] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.872437 2884 raft_consensus.cc:515] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.872720 2946 tablet_loader.cc:96] loaded metadata for tablet 140667afe42f42659a8cd38ed2517201 (table default.Table_13e60d7ade27439bbcab22ee125a50ba [id=8789a018b1c24bc3956fd11aea52b20e])
I20251214 20:52:35.873041 2884 leader_election.cc:290] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.873457 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bed2c326906486e99300e05033c9c0f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.873605 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8deabfdbd0674b2884ee5f82fa5e332f" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.879814 2946 tablet_loader.cc:96] loaded metadata for tablet 144dee1a6d7344f2b79da6e79e67599e (table default.Table_7306395aef6240c39e095e0c8815130b [id=8cd62da3d57c4be4b5f10ace49d2e148])
I20251214 20:52:35.879917 1203 tablet_bootstrap.cc:492] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.880251 2884 raft_consensus.cc:493] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.880402 2884 raft_consensus.cc:515] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.881289 2884 leader_election.cc:290] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.881572 2889 raft_consensus.cc:493] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.881914 1203 ts_tablet_manager.cc:1403] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.140s user 0.004s sys 0.003s
W20251214 20:52:35.882193 713 leader_election.cc:343] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 0d7c7b3f62764ff497f4d70b53eed7de
I20251214 20:52:35.882987 2946 tablet_loader.cc:96] loaded metadata for tablet 147c985fa7c94f7a9af57917bf17b29f (table default.Table_14f3cdeac59a4c20988d0e23744d2fbd [id=d53e0d2c6a7b4dc89f93ddd315732229])
I20251214 20:52:35.883687 2946 tablet_loader.cc:96] loaded metadata for tablet 15b3216f3d2b443e8ab2f994445956f9 (table default.Table_9db4b4a52f48436eaa4d06ee1e411237 [id=f189601a06e74bb38acdf3c0350430e8])
I20251214 20:52:35.884371 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d7c7b3f62764ff497f4d70b53eed7de" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.884889 2889 raft_consensus.cc:3060] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.884950 2884 consensus_queue.cc:1048] T fc7353593bda4fceaa389c393ddc19c6 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.885541 2946 tablet_loader.cc:96] loaded metadata for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e])
W20251214 20:52:35.886313 848 leader_election.cc:343] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.889709 2946 tablet_loader.cc:96] loaded metadata for tablet 185049f1ef43499aa1d15d99549255f7 (table default.Table_3b0a7d868e794df9bf7ec67f8b299c35 [id=0e2f32942b8d49c68d9fd9212e0702aa])
I20251214 20:52:35.891033 2946 tablet_loader.cc:96] loaded metadata for tablet 19008711a4754ca7b38a3ef2f0cdbf1d (table default.Table_a07fe706644d4c63b966f0751b757642 [id=8845cef019b44676a83a3173c5df0bb3])
I20251214 20:52:35.891479 2946 tablet_loader.cc:96] loaded metadata for tablet 190afbb866854dd7b5393e04443752b7 (table default.Table_501d1ae22d314c82914be72476d1ee0d [id=1f1f307f926f4ce297cf3f9c5423e818])
I20251214 20:52:35.897038 2777 raft_consensus.cc:493] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.897405 903 raft_consensus.cc:2468] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:35.897495 2777 raft_consensus.cc:515] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.897439 3108 raft_consensus.cc:493] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.898626 712 leader_election.cc:304] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:35.898916 2884 raft_consensus.cc:2804] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.898983 2884 raft_consensus.cc:493] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.899046 2884 raft_consensus.cc:3060] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.900092 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f9def8e4fd449599820a63845397750" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.900799 848 leader_election.cc:343] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.901572 2946 tablet_loader.cc:96] loaded metadata for tablet 197d3d340f6e4cbea0d621907b8c76d4 (table default.Table_f4c54cf75bea445185ff40f822983fb5 [id=12f54ade07c6481298fe1a6783f719d4])
I20251214 20:52:35.901772 2884 raft_consensus.cc:515] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.902961 1203 raft_consensus.cc:359] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:35.903311 713 leader_election.cc:343] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 0d7c7b3f62764ff497f4d70b53eed7de
I20251214 20:52:35.903399 1203 raft_consensus.cc:385] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.903605 1203 raft_consensus.cc:740] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.904002 2884 leader_election.cc:290] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.904150 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d7c7b3f62764ff497f4d70b53eed7de" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:35.904405 903 raft_consensus.cc:3060] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.904615 2946 tablet_loader.cc:96] loaded metadata for tablet 1986f856a8b44d62919976cd9c6b5f34 (table default.Table_26fdf6b490ab4f6cb488f406406a190c [id=d64c201020c34baf8cf82672f00aa29e])
I20251214 20:52:35.905010 2946 tablet_loader.cc:96] loaded metadata for tablet 1afeb64843e64f5b9b45259b1a3ec3d8 (table default.Table_1dc40537749846a8a77948884d646257 [id=ae41e2897b3a4deda0dacb28baa72adf])
I20251214 20:52:35.905666 2946 tablet_loader.cc:96] loaded metadata for tablet 1c82068013e8471c9309bcf8e8604ece (table default.Table_8bb698451aaf41b3a5c13aba022bbb57 [id=15c2e95c6dd24236b70614df50c6218c])
I20251214 20:52:35.905731 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8deabfdbd0674b2884ee5f82fa5e332f" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:35.906147 3108 raft_consensus.cc:515] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.907402 712 leader_election.cc:304] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:35.907512 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e185d107619414a8bcc15ad496547dc" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.907706 1050 raft_consensus.cc:2468] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.907624 2946 tablet_loader.cc:96] loaded metadata for tablet 1dc5e36369574906a0602145b37aa8d6 (table default.Table_da61d1abe3f24498977b6cc43d2ed4df [id=800f51296fe14db2985b673976f6ec6c])
I20251214 20:52:35.908139 851 leader_election.cc:304] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:35.908578 2884 raft_consensus.cc:2749] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:35.908923 2777 leader_election.cc:290] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.909632 2946 tablet_loader.cc:96] loaded metadata for tablet 1ee28c4ac91448629c0bddb3143f7df0 (table default.Table_bf1d2eb618b64a4e86cd121824fec0cb [id=1c14038af47c43c28ec8370fba0a14df])
I20251214 20:52:35.910552 2946 tablet_loader.cc:96] loaded metadata for tablet 2021de48ee904cceb2b5e49131164689 (table default.Table_da9e4432a9254052ace01a09e425f97b [id=6203767083704e8b818e367287cfb130])
I20251214 20:52:35.910598 2889 raft_consensus.cc:515] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.910929 2946 tablet_loader.cc:96] loaded metadata for tablet 202b1d53cab644c98838b0ee81f29f22 (table default.Table_cddd59a230e04595bffdf9e0a0ed1471 [id=6094596f6c064c5bb1e20aa2b8767b9f])
I20251214 20:52:35.911326 2889 leader_election.cc:290] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.911509 2946 tablet_loader.cc:96] loaded metadata for tablet 205e951fef7445d79a900f256cff8074 (table default.Table_2541d743bd8945a2b0d984cdf911d6ba [id=825f8141b3ca4a1983f6548c66239504])
I20251214 20:52:35.911635 1206 tablet_bootstrap.cc:654] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.912091 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bed2c326906486e99300e05033c9c0f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.912655 2946 tablet_loader.cc:96] loaded metadata for tablet 234a3f2c04e94149a1aeca848e6a4ce4 (table default.Table_bdfed01cc4cd4458be399de18a738f45 [id=164c3a6291b644789ae5905faca45b10])
I20251214 20:52:35.910339 1203 consensus_queue.cc:260] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:35.913147 2946 tablet_loader.cc:96] loaded metadata for tablet 2398f176ee044c15841eaa40cb74974c (table default.Table_25630f29e4904aaf86f8f65078bcf81e [id=b1ee89b437fb42a4b5e2c3d386ee0284])
I20251214 20:52:35.913498 3108 leader_election.cc:290] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:35.913511 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e185d107619414a8bcc15ad496547dc" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:35.914109 3108 raft_consensus.cc:2804] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.914196 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f9def8e4fd449599820a63845397750" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:35.914319 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bed2c326906486e99300e05033c9c0f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.914413 1056 raft_consensus.cc:2468] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:35.914449 3108 raft_consensus.cc:493] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.914503 1050 raft_consensus.cc:3060] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.914534 2946 tablet_loader.cc:96] loaded metadata for tablet 243528f1aeac425c8be47f9fae802f52 (table default.Table_8923490b824a47139d8f739d65e4845b [id=2ffffc8a53e04583a2f93b287fe083bf])
I20251214 20:52:35.914561 3108 raft_consensus.cc:3060] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.916057 851 leader_election.cc:304] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
W20251214 20:52:35.916587 848 leader_election.cc:343] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.918332 1050 raft_consensus.cc:2468] T 3bed2c326906486e99300e05033c9c0f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.919145 2777 raft_consensus.cc:2804] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:35.919265 2777 raft_consensus.cc:493] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:35.919312 2946 tablet_loader.cc:96] loaded metadata for tablet 25180d5352204cdca3c49f907a587cb2 (table default.Table_ca7b705bd1754ce19fbd7cda96364054 [id=525aea874ba84a65af4b2cbbed3fbf9e])
I20251214 20:52:35.919358 2777 raft_consensus.cc:3060] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.919677 2946 tablet_loader.cc:96] loaded metadata for tablet 2593f67c7c6441b694d99286e3ba5883 (table default.Table_bde8bb1617d3450f999e9a7960d96641 [id=3a3dca8b90264d949ac9bead0b0743fc])
I20251214 20:52:35.920310 2946 tablet_loader.cc:96] loaded metadata for tablet 259762515ce44ca6b7f4cd855310d1ec (table default.Table_e7a205677f614917bc69ca73b5fb54da [id=262d9252040b4138ad42d1c05a402cd7])
I20251214 20:52:35.920538 2889 consensus_queue.cc:1048] T e59b80ee919b4e1393fe317ed0e3a3d2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
W20251214 20:52:35.921010 848 leader_election.cc:343] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.920850 1203 ts_tablet_manager.cc:1434] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.038s user 0.003s sys 0.000s
I20251214 20:52:35.921164 2946 tablet_loader.cc:96] loaded metadata for tablet 2807af0754914688bd7a090b1bc32eb1 (table default.Table_c6c0496d455a4db68c05c0ff8b06493d [id=9d1473df58ee486aa6a55893aca9d54c])
I20251214 20:52:35.921809 2946 tablet_loader.cc:96] loaded metadata for tablet 28d34670fdce4242a3feccf830a12880 (table default.Table_4d5d73b55b6245bf8594a4fc79f02adc [id=8316131458234291a566634324832ffb])
I20251214 20:52:35.921792 851 leader_election.cc:304] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:35.923477 3141 raft_consensus.cc:2804] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:35.924239 2946 tablet_loader.cc:96] loaded metadata for tablet 2d5920fdba794e5f864c595e7be278ab (table default.Table_873207e85b954e21b3d98ad4c02a0248 [id=451df68541d848278bfeda2dfce0ba7f])
I20251214 20:52:35.924726 3141 raft_consensus.cc:697] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:35.924960 2946 tablet_loader.cc:96] loaded metadata for tablet 2ea3c1e1ca5a41df8a01546bc08c3e23 (table default.Table_80391b5a5aa9425794dfb7997c659009 [id=893edd42fa04409b9d8e4a4e46d38ebc])
I20251214 20:52:35.925033 3141 consensus_queue.cc:237] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.925395 2946 tablet_loader.cc:96] loaded metadata for tablet 2ef81c05366743d28579465e95a79293 (table default.Table_e2a035220b924122a538db32c77a683b [id=f97b0be6362b47b49c15807bdf090e66])
W20251214 20:52:35.927383 848 consensus_peers.cc:597] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.927502 2946 tablet_loader.cc:96] loaded metadata for tablet 30595c2e1f814a66b29722caf123edce (table default.Table_db687aab85f2411f8433eb14724da9ac [id=83ef50f71fb34ecda09e60f00f8ec8a7])
I20251214 20:52:35.928370 2946 tablet_loader.cc:96] loaded metadata for tablet 336cb9d03fd44f70930662ee3c8d987a (table default.Table_ebccdf7648384d44b9629a1b41509e68 [id=2e1e99c74ef5467190886342facb1a5b])
I20251214 20:52:35.928771 2946 tablet_loader.cc:96] loaded metadata for tablet 3373921f73b64fae954ea7c5d5f1d93f (table default.Table_5cda3bedf2d24fc2b279c049146aa2c5 [id=10ef2e196edd432aa21fd7d31fa4366f])
I20251214 20:52:35.929255 2946 tablet_loader.cc:96] loaded metadata for tablet 33a7f355b40f472883e3399209ef6762 (table default.Table_12d421ed905d41eead75b19eb8a793b6 [id=5272550343144d4baaba07e97ebaa187])
I20251214 20:52:35.934341 1203 tablet_bootstrap.cc:492] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:35.935184 1206 tablet_bootstrap.cc:492] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:35.937680 1206 ts_tablet_manager.cc:1403] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.004s
W20251214 20:52:35.942744 848 consensus_peers.cc:597] T 8e11723c042a4637839f1b13bed0935a P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.945740 2777 raft_consensus.cc:515] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.947198 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f9def8e4fd449599820a63845397750" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:35.947386 1051 raft_consensus.cc:3060] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:35.949612 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f9def8e4fd449599820a63845397750" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.951908 1051 raft_consensus.cc:2468] T 2f9def8e4fd449599820a63845397750 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:35.952965 3039 consensus_queue.cc:1048] T fe24e4f4e74040249068505058c15622 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:35.954824 2777 leader_election.cc:290] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:35.952432 3119 raft_consensus.cc:493] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:35.958738 1203 tablet_bootstrap.cc:654] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:35.960801 3119 raft_consensus.cc:515] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.961545 3119 leader_election.cc:290] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:35.961591 2946 tablet_loader.cc:96] loaded metadata for tablet 365dc8e0324c415fb35d857849911595 (table default.Table_c96142a94af44f24bb7c6e51bf5e1f3e [id=296657ce558f4c599fc2013a71b7bda3])
I20251214 20:52:35.964828 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "880a78b924574181bdd804d599f4f579" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:35.965442 989 leader_election.cc:343] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:35.966230 848 leader_election.cc:343] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.966727 3108 raft_consensus.cc:515] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.967061 903 raft_consensus.cc:2468] T 0d7c7b3f62764ff497f4d70b53eed7de P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:35.968151 2946 tablet_loader.cc:96] loaded metadata for tablet 37d198fa8da642dca442476578a35f1e (table default.Table_81b328890d6748ea8a95ac3cd696c9dd [id=4a964e807c8a4161bf9fa9cb732ba21d])
I20251214 20:52:35.968719 2946 tablet_loader.cc:96] loaded metadata for tablet 38d688311da24b5cac50fe10ed9e6ab8 (table default.Table_81e31e2b08a1484c8adb4f139e0edfd3 [id=f6a2470a04cf40e0b3d04ec337fe328b])
I20251214 20:52:35.969412 2946 tablet_loader.cc:96] loaded metadata for tablet 3a0f669313244e809c2868ca0ccfad65 (table default.Table_d59e2b0d82384da68c1901ddaa419d00 [id=0c56f14b40ac442994e232aa348d64a2])
I20251214 20:52:35.969848 2946 tablet_loader.cc:96] loaded metadata for tablet 3a23561b9f4e48bbbd796e14fafec086 (table default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3 [id=f1338e81f4ec4dce87130f2ddbe98041])
I20251214 20:52:35.971902 2946 tablet_loader.cc:96] loaded metadata for tablet 3de3864300a94e53a856c6a93e8d6e80 (table default.Table_fc742d049909446bbdb1a0b0973734c0 [id=233d035cea634e6a82f2505aa9bf0e32])
I20251214 20:52:35.973168 1203 tablet_bootstrap.cc:492] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:35.974160 2946 tablet_loader.cc:96] loaded metadata for tablet 3f465cef46cc4e23982dced0d2ea9e36 (table default.Table_70cb422b9a3d439cba4cddcdd9b31c73 [id=74eb42fa8016492cad785dfa4d0088c9])
I20251214 20:52:35.973994 1203 ts_tablet_manager.cc:1403] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.000s
I20251214 20:52:35.974603 2946 tablet_loader.cc:96] loaded metadata for tablet 3fa0317e999b483788e5c5652e6e83de (table default.Table_3c676f861c28408f99faac2ab8f7eec1 [id=ef22180b36a241fea13d0e622c02b50a])
I20251214 20:52:35.975554 2946 tablet_loader.cc:96] loaded metadata for tablet 4106fb80b4304735bc3faba7a7bf342f (table default.Table_b567effdb6834f7b87ce4aba9e555913 [id=c216b38d4fce484da23d096493ac2934])
W20251214 20:52:35.976022 848 consensus_peers.cc:597] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:35.977962 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e185d107619414a8bcc15ad496547dc" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:35.982549 1203 raft_consensus.cc:359] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.983606 3108 leader_election.cc:290] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:35.983546 1203 raft_consensus.cc:385] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:35.983883 1203 raft_consensus.cc:740] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:35.984974 1203 consensus_queue.cc:260] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:35.985730 2946 tablet_loader.cc:96] loaded metadata for tablet 41effce1ac504f27b619a4d9950202f9 (table default.Table_0fc25951e7cc4aebac50ad75316773f5 [id=d4196eff6f0c4965bfb9aa723570f6f3])
W20251214 20:52:35.987489 848 leader_election.cc:343] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:35.996307 1203 ts_tablet_manager.cc:1434] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20251214 20:52:35.999300 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "880a78b924574181bdd804d599f4f579" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.001915 2946 tablet_loader.cc:96] loaded metadata for tablet 42bdcecba5534294aca1eb419f18e9d2 (table default.Table_0fd459d9720046cc9ff44aaf767d2994 [id=b3fbe031a2e645c4aa0754a1452d62c8])
I20251214 20:52:36.002807 1206 raft_consensus.cc:359] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.003026 1206 raft_consensus.cc:385] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.003085 1206 raft_consensus.cc:740] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.003700 2946 tablet_loader.cc:96] loaded metadata for tablet 438b8ec5e9834442994966c8d83c1a31 (table default.Table_5cffb2d16a4847caa5cf6a2c4c238da8 [id=a8b3bbe26fc14b4bbc8a78c3d1f9df6d])
I20251214 20:52:36.005039 2946 tablet_loader.cc:96] loaded metadata for tablet 4408013cf49e46d0b18d24b48483ea49 (table default.Table_7d6d418a1acb404f908e9ee299cef249 [id=55fc2a6c5e514513904682dfde510d9f])
I20251214 20:52:36.006738 2946 tablet_loader.cc:96] loaded metadata for tablet 4501fd5566a34e2a82009cc9ce6bdd37 (table default.Table_dce91ff3d9c74c7db1d400a24c4e550a [id=14f12b293a274ebeaf5bfefdff5b78cf])
I20251214 20:52:36.008294 2946 tablet_loader.cc:96] loaded metadata for tablet 472825da1a5243ab984f393ce98c8fe9 (table default.Table_eaeae6c30719483681416f2d216d1209 [id=68304e9495f44caab2e47d46335ab30f])
I20251214 20:52:36.009475 1202 tablet_bootstrap.cc:492] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:36.010187 2884 raft_consensus.cc:493] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.010356 2884 raft_consensus.cc:515] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.010788 2946 tablet_loader.cc:96] loaded metadata for tablet 47ca12fc0de44082aa3d4fabc919c1dc (table default.Table_02bb72c146234757b420555977074934 [id=5f186d0854254ce287edb0ccb77ee6b7])
I20251214 20:52:36.010950 2884 leader_election.cc:290] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:36.011942 713 leader_election.cc:343] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 69f6108792fa4c2a9744e3248c76251f
I20251214 20:52:36.012594 1206 consensus_queue.cc:260] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.012651 851 leader_election.cc:304] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.015316 1203 tablet_bootstrap.cc:492] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.019409 2889 raft_consensus.cc:2804] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
W20251214 20:52:36.019724 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 57213.4us
I20251214 20:52:36.021039 712 leader_election.cc:304] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:36.021106 1203 tablet_bootstrap.cc:654] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.021692 2884 raft_consensus.cc:2804] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.021806 2884 raft_consensus.cc:697] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:36.022014 2884 consensus_queue.cc:237] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:36.024436 712 leader_election.cc:343] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 69f6108792fa4c2a9744e3248c76251f
I20251214 20:52:36.024562 712 leader_election.cc:304] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.024895 3039 raft_consensus.cc:2749] T 69f6108792fa4c2a9744e3248c76251f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.028249 2889 raft_consensus.cc:697] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.028565 2889 consensus_queue.cc:237] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.031514 1202 ts_tablet_manager.cc:1403] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.181s user 0.010s sys 0.010s
I20251214 20:52:36.033767 1202 raft_consensus.cc:359] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.034020 1202 raft_consensus.cc:385] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.034128 1202 raft_consensus.cc:740] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.034379 1202 consensus_queue.cc:260] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.034674 2946 tablet_loader.cc:96] loaded metadata for tablet 47e873c6be06401282f60d87d1a70865 (table default.Table_b58a6dfd93fb438684f7564325241f3e [id=e85a8088178744e0a19699f5d680b4fa])
I20251214 20:52:36.037039 1202 ts_tablet_manager.cc:1434] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251214 20:52:36.037552 2946 tablet_loader.cc:96] loaded metadata for tablet 47f610e6d08046a385c58274abab0e0b (table default.Table_2be03ca8d3964e4db04c188da2e59bd3 [id=1eeba8e8c0be4ac2b904a59363a89e99])
I20251214 20:52:36.039616 1202 tablet_bootstrap.cc:492] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:36.040185 848 consensus_peers.cc:597] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:36.040838 2946 tablet_loader.cc:96] loaded metadata for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 (table default.Table_403e5819717949e8bffa1ed844a68964 [id=faaedb3e871c491691ecec42c3ab9d8f])
I20251214 20:52:36.043639 2946 tablet_loader.cc:96] loaded metadata for tablet 489a43e7a1da41f4b1a8e80198eec212 (table default.Table_4cb054a62c404f1a952f39810b6d12f8 [id=515eec002ab94ec396c6bc70dae4d996])
I20251214 20:52:36.044461 1202 tablet_bootstrap.cc:654] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.048336 3226 raft_consensus.cc:493] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251214 20:52:36.048491 991 leader_election.cc:343] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.048636 2946 tablet_loader.cc:96] loaded metadata for tablet 49778cfe80f6483389cfd421abee8ca7 (table default.Table_3024ac839d9441db89239acebb00bc12 [id=d3229230c8844d2fb42cd81149462a40])
I20251214 20:52:36.048662 991 leader_election.cc:304] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.048636 3226 raft_consensus.cc:515] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.049162 3119 raft_consensus.cc:2749] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.049288 2946 tablet_loader.cc:96] loaded metadata for tablet 4a625fcb95324466a12ab118bdb859f1 (table default.Table_ac87176ddd0249279bffb87faebdab72 [id=0502b4d977444acda3f1a5f7ff69d974])
I20251214 20:52:36.049674 2946 tablet_loader.cc:96] loaded metadata for tablet 4a6c43ece40349fcb6c875c8b6d49409 (table default.Table_f86bf75508ca43ada4372af83098ae35 [id=ecd6b2f07b474967adf0662a8b89979b])
I20251214 20:52:36.050299 2946 tablet_loader.cc:96] loaded metadata for tablet 4b5360e7ece94fd2a5a6512db6c6c12f (table default.Table_2ca54bf356e24a1ebca6f4a155b83044 [id=7f407f5abea945498cf939a23b0f9319])
I20251214 20:52:36.050662 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d5920fdba794e5f864c595e7be278ab" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.050647 3226 leader_election.cc:290] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:36.051265 848 leader_election.cc:343] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.051394 1203 tablet_bootstrap.cc:492] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.051757 1203 ts_tablet_manager.cc:1403] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.000s
I20251214 20:52:36.051990 2946 tablet_loader.cc:96] loaded metadata for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e])
I20251214 20:52:36.052377 2946 tablet_loader.cc:96] loaded metadata for tablet 4e4dc66681ef48479ae3cb95b091564a (table default.Table_00ec2fefc3d84a8b9f94bf9d194e2045 [id=12e7c17407a444dd8e0c0ad40fb89e28])
I20251214 20:52:36.053139 2946 tablet_loader.cc:96] loaded metadata for tablet 4f1d0e555eee4c56be271d14c34335bd (table default.Table_180ac2f089ba44298206726a47570c5b [id=81609d89fec84b2f9b79ca8986447b65])
I20251214 20:52:36.053534 2946 tablet_loader.cc:96] loaded metadata for tablet 4f22ac24c38e4c64bb26c54ae4d87dfb (table default.Table_b2ac4ca26b6640a287e7f3b6974c2363 [id=15ed55ac8a0249bfbc08076265293bd6])
I20251214 20:52:36.053381 1203 raft_consensus.cc:359] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.053797 1203 raft_consensus.cc:385] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.053951 1203 raft_consensus.cc:740] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.054214 2946 tablet_loader.cc:96] loaded metadata for tablet 51366a6aefd54077877ec5d076394d79 (table default.Table_1191163db85341c5b41c4baf61bc8df5 [id=fcb196b8ebb14e1785ea355e7d406b8a])
I20251214 20:52:36.054608 2946 tablet_loader.cc:96] loaded metadata for tablet 5157ef5f1fd541efbbd2887bf4505254 (table default.Table_d80c747f5a704a608b5e28b5e5d247b1 [id=e09da957fd5b4879a87ffb8c1cdd4bbb])
I20251214 20:52:36.054440 1203 consensus_queue.cc:260] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.054997 2946 tablet_loader.cc:96] loaded metadata for tablet 526f28f9aaad445081d9d69cc20ed7b3 (table default.Table_b87980e2da6a4052b109938fcf0527d8 [id=88d5807cc76646e3a2962b5d36413894])
I20251214 20:52:36.056668 2946 tablet_loader.cc:96] loaded metadata for tablet 52f5af48440b4d569939c163f365f6fe (table default.Table_52200ef9aa154f64ae25af61268f6d6b [id=35d3089ba6ef4f28ada7749dc8fb5ceb])
I20251214 20:52:36.057097 2946 tablet_loader.cc:96] loaded metadata for tablet 532c44785f204f988518c6fd628f76ca (table default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39])
I20251214 20:52:36.057690 2946 tablet_loader.cc:96] loaded metadata for tablet 556ac6b97fab4ea5a43089c0f96c38e5 (table default.Table_8f5d31355346492989caa148a8f1440b [id=ec2fd20884ff4bdca7be9d32d3bfe9ae])
I20251214 20:52:36.058054 2946 tablet_loader.cc:96] loaded metadata for tablet 557fcbbc7889434e87ca82177c5d0eb0 (table default.Table_7b56d13e748e405a821ea513405846fa [id=2e995b9d47b04d3d8f5a01482c57d0d2])
I20251214 20:52:36.058430 2946 tablet_loader.cc:96] loaded metadata for tablet 566c6e4761d5486d964e276d151e32b0 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615])
I20251214 20:52:36.060153 2946 tablet_loader.cc:96] loaded metadata for tablet 59ccb6c3b35044dc913cce74b5802e38 (table default.Table_d61784eed0b84f8caa485dbe70cdc366 [id=b5622661ea9f40f7ac11f44ae92a5194])
I20251214 20:52:36.060556 2946 tablet_loader.cc:96] loaded metadata for tablet 5a43ee234b184275a6ab76b82fe5d283 (table default.Table_0a325ff1f4634ce7ada593df8f5a9fd6 [id=88a6486d16184397acd4ef205ca46a7a])
I20251214 20:52:36.060936 2946 tablet_loader.cc:96] loaded metadata for tablet 5afb7fb4f0a44b0092fbef5bbec9f396 (table default.Table_87f9c82a6a89490391b5c78001639d2b [id=8203de75aba94c3f8561f18adaaa825e])
I20251214 20:52:36.061295 2946 tablet_loader.cc:96] loaded metadata for tablet 5b1576eff7e94f02a373e02f8e4a961d (table default.Table_529fb2317e754462b7ae49d9304e5402 [id=9187b59d5c7949169b104f586e7edad7])
I20251214 20:52:36.062139 3119 raft_consensus.cc:493] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.062294 3119 raft_consensus.cc:515] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.062373 2946 tablet_loader.cc:96] loaded metadata for tablet 5e92f54daf4f495fb6f026c179852362 (table default.Table_d6683a16a95f4274b82ff74dc7e73597 [id=573bdb65191b4f09b43190844f2b0008])
I20251214 20:52:36.062681 2946 tablet_loader.cc:96] loaded metadata for tablet 5ec7824f5eff4cc59c45acd3b3b3d194 (table default.Table_6cfa43d70d2f426fb58f8ef32476c3ac [id=be1be47d388347e3a6b7a80105688be3])
I20251214 20:52:36.062987 3119 leader_election.cc:290] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.063040 2946 tablet_loader.cc:96] loaded metadata for tablet 5f251ff9b3fa4fe8ababab456733bcdd (table default.Table_b9ef85d4768d42d28d635a7c3eac5f4c [id=7be3dc7e56824beb9f13c242645bdff4])
I20251214 20:52:36.064532 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cea9423c40044e4cabb8bbd755a46688" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.064924 2946 tablet_loader.cc:96] loaded metadata for tablet 5f6be4dbadc74f468fe5f92ecf158e63 (table default.Table_47682ee6f46147558b5033379c2fe733 [id=788e2409150b408996fcc9e70c026bd0])
W20251214 20:52:36.065173 989 leader_election.cc:343] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.066866 2991 consensus_queue.cc:1048] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:36.076143 848 consensus_peers.cc:597] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:36.078285 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cea9423c40044e4cabb8bbd755a46688" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.079924 1206 ts_tablet_manager.cc:1434] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.142s user 0.002s sys 0.000s
W20251214 20:52:36.081449 848 consensus_peers.cc:597] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:36.082283 991 leader_election.cc:343] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.082448 991 leader_election.cc:304] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.082921 3119 raft_consensus.cc:2749] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.085266 1206 tablet_bootstrap.cc:492] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:36.085916 1203 ts_tablet_manager.cc:1434] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20251214 20:52:36.090523 2946 tablet_loader.cc:96] loaded metadata for tablet 62ac29a23c174ad9a3d93a10af7fecbe (table default.Table_8ae98ffc7458410786e7844521f3c5c1 [id=4c6dacf190dd4c7aa5b4e056bf7dfd8c])
I20251214 20:52:36.091187 2946 tablet_loader.cc:96] loaded metadata for tablet 6302145a6e7b4297a58d9d7f3ad02dce (table default.Table_1be39fb576af4b2b9e2606b495619863 [id=54e0182cd3b9461aa593a11aa7709002])
I20251214 20:52:36.096055 2946 tablet_loader.cc:96] loaded metadata for tablet 6343dbb0658c4145a816c444ff5297d1 (table default.Table_8d0c148da21f4137b09c9631e0b3f9bf [id=fa40006b630d491e88479cd9f61def2e])
I20251214 20:52:36.096112 1203 tablet_bootstrap.cc:492] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.096760 2946 tablet_loader.cc:96] loaded metadata for tablet 636c209967fd4fe78d91d9e7a6fbf613 (table default.Table_62fb3a07f4a04029a6080d9d0e35d151 [id=c2886db5a22c43b5b371935e9f47187a])
I20251214 20:52:36.097376 2946 tablet_loader.cc:96] loaded metadata for tablet 639cf3b7654a4ae896c6edd8b2ff1844 (table default.Table_15c1781afe3449ef87ef3c655ccca0df [id=02c00037bc944c00b0124d72af62e19b])
I20251214 20:52:36.097975 2946 tablet_loader.cc:96] loaded metadata for tablet 6404321f391944769986905a12fbab1c (table default.Table_4abdf07a08814d65acc20b6c32bfd415 [id=aeb3160d155949699e454f389fd07aa3])
I20251214 20:52:36.098345 2946 tablet_loader.cc:96] loaded metadata for tablet 641a9073c73d41c09e07cabc60735ba4 (table default.Table_fd97b3b527e240fbb6d62d3c33059904 [id=c1ab5c38b5ee4129a8bccf874894ddcf])
I20251214 20:52:36.098986 2946 tablet_loader.cc:96] loaded metadata for tablet 6583412e5a1849c8b93c87ba7fdc4647 (table default.Table_3474e4ca80964e3b8b7b66e1fde4815f [id=586a58c36e9a4d4494f4538ebcb9ce36])
I20251214 20:52:36.099419 2946 tablet_loader.cc:96] loaded metadata for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7])
I20251214 20:52:36.100541 2946 tablet_loader.cc:96] loaded metadata for tablet 65b2fe0de22646e4b77fbc6a289ff4a1 (table default.Table_a426957d72cc493393637b00a3b1e12d [id=f2fc3f9c504f44cc9c8bc71d2e1c8cde])
I20251214 20:52:36.101289 2946 tablet_loader.cc:96] loaded metadata for tablet 67104e34df294ef5986171eca79cdc72 (table default.Table_27222912e9214b879d8a30975aba5688 [id=c0729ac770394d7b87ef36e53f85c151])
I20251214 20:52:36.102217 2946 tablet_loader.cc:96] loaded metadata for tablet 69f9d5838e544002ab49f0ce21b6f7ef (table default.Table_9ff099dcfac14636a9dcaa70207c86a6 [id=4a1816b57ff84e459394e59b683d8ec6])
I20251214 20:52:36.102813 2946 tablet_loader.cc:96] loaded metadata for tablet 6b384bbc1f874d758dadd6932470713c (table default.Table_5b82f109385a42b0a04a5272bc5f0156 [id=40f4f5bacfea4828880fc526be3a06cb])
I20251214 20:52:36.103536 2946 tablet_loader.cc:96] loaded metadata for tablet 6b482043af42485eb4c73d6eb864a963 (table default.Table_b5f628ed069a4b19996c9b579d7678bc [id=a03c04d94997449bac7447ed634577d4])
I20251214 20:52:36.104187 2946 tablet_loader.cc:96] loaded metadata for tablet 6bd280c0d8f94893ba8eb1e97bb89cb0 (table default.Table_f6c14f9b22304e8aafbe1afab658ae71 [id=b6efd20d2ef54f799f88337bfc4f7980])
I20251214 20:52:36.104864 2946 tablet_loader.cc:96] loaded metadata for tablet 6be0a51f763b474093cc7d08ea75411e (table default.Table_5126357c033645d1ae06a909124ab640 [id=72c5a8ce76cb46d89b7a563f8b16abc3])
I20251214 20:52:36.105286 2946 tablet_loader.cc:96] loaded metadata for tablet 6bf7317683ff4a52b3503687f4131ad3 (table default.Table_6588026e2d974db9b72d2b81ce8310cb [id=483610ffc7db4e848d0756374015974f])
I20251214 20:52:36.106015 2946 tablet_loader.cc:96] loaded metadata for tablet 6c1893b3761c49ceb7cd71b3bd9f2cb4 (table default.Table_250b3222a83c44b08856444499fa612d [id=1ca50be529bd48398643cb6882958f77])
I20251214 20:52:36.106451 2946 tablet_loader.cc:96] loaded metadata for tablet 6c8af3d2720a499eb51bcfb213bcdbd1 (table default.Table_ae25c010c04548198844cd6f5f3a5693 [id=7159a40b641e406d89e8a11b2b1eba82])
I20251214 20:52:36.106858 2786 raft_consensus.cc:493] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.107542 2786 raft_consensus.cc:515] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.108906 2786 leader_election.cc:290] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.109431 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57a6232564194cbba12182f3a29c2ebb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:36.110011 848 leader_election.cc:343] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.110534 2906 raft_consensus.cc:493] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.110673 2906 raft_consensus.cc:515] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.111294 2906 leader_election.cc:290] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.107095 2946 tablet_loader.cc:96] loaded metadata for tablet 6cd04f871edb4fb78d6ec852c062ef54 (table default.Table_6fedc617de7e4cd985c616534bda854a [id=5c2a10eb6dca403a94f387331ec8c3d4])
I20251214 20:52:36.112375 2946 tablet_loader.cc:96] loaded metadata for tablet 6e005feeb95b48b9ad33612ab81d5bce (table default.Table_ec8c8f9185be4470a6580c4e141e2227 [id=8872bfac95c64cbba1ccb84d1d9f2483])
I20251214 20:52:36.112991 2946 tablet_loader.cc:96] loaded metadata for tablet 6e8289d6605d4d35a4a475395a23d5e1 (table default.Table_c50d34acaf2e4ec8be9f4c60d90ec25c [id=2605ab6b89da49229adbc897734da196])
I20251214 20:52:36.113209 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1def5f0de5ce4eb0bed36f626c4f0683" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.114192 2946 tablet_loader.cc:96] loaded metadata for tablet 6f231db5cc4a4060a996465defb23c6c (table default.Table_e277c40da9e24b7fabb8edd25ec3413d [id=e973536efad243e5b94f5994d57325b4])
W20251214 20:52:36.114354 989 leader_election.cc:343] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.114639 2946 tablet_loader.cc:96] loaded metadata for tablet 6fef1d6add764fd28a8f4995b337ea71 (table default.Table_3385c79468e4468eb0cddb80a5cf0c75 [id=f4631274e1b44753a96faa02c9f13980])
I20251214 20:52:36.115048 2946 tablet_loader.cc:96] loaded metadata for tablet 700ed62ef2e84719a7a8e9879feaf45c (table default.Table_65d3039202a645e899e5f752e1e3db42 [id=b6bebe3ab7e2449dafc61fdda3e3c38c])
I20251214 20:52:36.115466 2946 tablet_loader.cc:96] loaded metadata for tablet 707393e4f91e4b1dbe1cf7485fa65624 (table default.Table_17c9057bb25b4cce9b1072a93f6ea38a [id=5727ded20a714c31be1ba62f2dfb21e4])
I20251214 20:52:36.115911 2946 tablet_loader.cc:96] loaded metadata for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_9060872cac6842548b9f1f80d16dbe19 [id=b0f347532cc84fcca5892f5a2443321c])
I20251214 20:52:36.116170 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1def5f0de5ce4eb0bed36f626c4f0683" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.116277 2946 tablet_loader.cc:96] loaded metadata for tablet 71b8588773c442fa9add357455fc9d7d (table default.Table_4278bf82042b47c2b4d0e799f115c52f [id=2c037ca94c8a4c908d6354f2aa802bf6])
W20251214 20:52:36.116767 991 leader_election.cc:343] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.116887 991 leader_election.cc:304] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.117169 2946 tablet_loader.cc:96] loaded metadata for tablet 7242be189ca54c9182181e33d6743e6b (table default.Table_f57172cc26534c9eafbefb6eb9a4d2e9 [id=23efc0ee36244ca0a6ff0f2faeb00b1f])
I20251214 20:52:36.117228 3119 raft_consensus.cc:2749] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.115739 3141 raft_consensus.cc:493] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.117846 2946 tablet_loader.cc:96] loaded metadata for tablet 737b767718c54512a1547262db13b060 (table default.Table_1f214ce7548c43af80a74e233714ca0d [id=8f83a24aa5834c4b8c8bec3f0a8ac29e])
I20251214 20:52:36.117630 3141 raft_consensus.cc:515] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.118355 2946 tablet_loader.cc:96] loaded metadata for tablet 738668adf2874d6894db6fa875f6990c (table default.Table_5e0ee6bb376f49bb925ed69ce3a82727 [id=406157031d694ada8a9fcba4e4e23e90])
I20251214 20:52:36.118774 2946 tablet_loader.cc:96] loaded metadata for tablet 74d2f8e5481d40b39cc4df34b0d4f819 (table default.Table_0fb8f39742d245ac9d20fc7d6bf7e7c3 [id=552c8b213ba64d4bbdd3363cdc6625ff])
I20251214 20:52:36.119627 2946 tablet_loader.cc:96] loaded metadata for tablet 76472ce9005b4bd2bbc51278816e8ae0 (table default.Table_9698b9206b934e4db2decdf9c0180838 [id=4b3f6c8d8cce4f7a96b2c677df6469ba])
I20251214 20:52:36.120348 2946 tablet_loader.cc:96] loaded metadata for tablet 76d16819e133434eb1c5a65fffb1f03f (table default.Table_0945dd902049447298aef35db32f33ba [id=3b87708d1c7b460d925f7da030a1a23f])
I20251214 20:52:36.120790 2946 tablet_loader.cc:96] loaded metadata for tablet 76e98dbef93c437c83db3b6dbdb06afa (table default.Table_58619bc0f4b54a208cabd270329ccf82 [id=3f4e5f296df7465b9c9ef0c86e7e2b76])
I20251214 20:52:36.121156 2946 tablet_loader.cc:96] loaded metadata for tablet 780be07e6f5d425d9605402b6db63012 (table default.Table_ee7a207364d74e6b81d6a72e57cacbd5 [id=b251a30885f6435fbfa430b6b50aee7a])
I20251214 20:52:36.121371 1202 tablet_bootstrap.cc:492] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:36.121383 1206 tablet_bootstrap.cc:654] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.121567 1202 ts_tablet_manager.cc:1403] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.003s
I20251214 20:52:36.121644 2946 tablet_loader.cc:96] loaded metadata for tablet 7947b48d8e2e423da187cadf176f38ce (table default.Table_7694eee14c2346a9a71802bdf0897011 [id=9f14a0a266b947559b2173b20ead5ccd])
I20251214 20:52:36.122756 2946 tablet_loader.cc:96] loaded metadata for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738 (table default.Table_912c9fc9392c4d99aa8485af7917eb85 [id=a56279b96c2d45af90277c7636192936])
I20251214 20:52:36.123240 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1eb4c3aa654480caaa69dbd5047365c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.123489 775 raft_consensus.cc:2468] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:36.123945 848 leader_election.cc:304] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:36.124512 2946 tablet_loader.cc:96] loaded metadata for tablet 7e3e5f83fe6044219b6e34d7b290211f (table default.Table_acd41f528acd483cb2c37531c45283d6 [id=6cdbb231eb7e4d01b6023731b39f16a4])
I20251214 20:52:36.124869 2786 raft_consensus.cc:2804] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.124980 2786 raft_consensus.cc:493] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.125056 2786 raft_consensus.cc:3060] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.125298 2946 tablet_loader.cc:96] loaded metadata for tablet 80098b7c24824d129b8f25daf2115460 (table default.Table_b14d897359c140bab6d04403fd59f31a [id=2a055be9028342be97f5c98f03f5a137])
I20251214 20:52:36.125774 1202 raft_consensus.cc:359] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.126019 1202 raft_consensus.cc:385] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.126114 1202 raft_consensus.cc:740] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.126336 2946 tablet_loader.cc:96] loaded metadata for tablet 80e28c4d87774256acb8e9638ded8689 (table default.Table_eec6b34ea20f4ef096ef8df379a3bddb [id=05a65b055df144b0bd3e83d8baba49a0])
I20251214 20:52:36.126550 1202 consensus_queue.cc:260] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.127015 2946 tablet_loader.cc:96] loaded metadata for tablet 812642cfcff940f899ef9d3644909ec4 (table default.Table_08a9cb7994804ae586d3a7e3de4adb7d [id=74f23f7675974c82953d757dd563179a])
I20251214 20:52:36.127952 1202 ts_tablet_manager.cc:1434] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251214 20:52:36.124289 3141 leader_election.cc:290] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:36.130759 2946 tablet_loader.cc:96] loaded metadata for tablet 827cddc5936346229e16611f0d73d587 (table default.Table_d050172b06964a17a220a95fb84d1081 [id=b2e03510ce354e1686aad6c081d52903])
I20251214 20:52:36.130672 1203 tablet_bootstrap.cc:654] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.131925 2991 raft_consensus.cc:493] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.132088 2991 raft_consensus.cc:515] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.132834 2991 leader_election.cc:290] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.135664 2946 tablet_loader.cc:96] loaded metadata for tablet 831d3bdd6d6845b9aa18d4cff52c2f0b (table default.Table_59b03580c6ff4f28baeb68e68994d7dc [id=0c71124d6937494cbded6debf9e9e4be])
I20251214 20:52:36.136345 2946 tablet_loader.cc:96] loaded metadata for tablet 83da479367fe4b949f55bc6756883892 (table default.Table_f7d781182d07441a824d887e07286dc3 [id=6170186dc9cc4f4e93394530eb7c3a56])
I20251214 20:52:36.137326 2946 tablet_loader.cc:96] loaded metadata for tablet 86ede5e41a03485082f4531fc42dfe56 (table default.Table_ee3c57e398fd468fa1e4e7cc22f6747a [id=ce8357dfd1024446a69a3ee196ab4b3f])
I20251214 20:52:36.137758 2946 tablet_loader.cc:96] loaded metadata for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (table default.Table_fcafcf3c6ddb4457aea41320f7bf0b74 [id=4329d01e7f6147d0adad53fa86369b17])
I20251214 20:52:36.138427 2946 tablet_loader.cc:96] loaded metadata for tablet 88b9ca42a5ce4572aebe2a24f4758b12 (table default.Table_adc1626f17674e3cab49c72818eec7d6 [id=0d173bfba2b343dbac6a02a5ff84c828])
I20251214 20:52:36.138818 2946 tablet_loader.cc:96] loaded metadata for tablet 894e3f27b7904db994711aa80a9b1600 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc])
I20251214 20:52:36.139523 2946 tablet_loader.cc:96] loaded metadata for tablet 899f21bafb3c4db9b23907cdc9f121d8 (table default.Table_e95c404ef11b4cf79b5a81ec1eddb57a [id=fb52e77ee3ec48c39afd588c8de11764])
I20251214 20:52:36.139570 2786 raft_consensus.cc:515] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.140169 2946 tablet_loader.cc:96] loaded metadata for tablet 89a0435db9a1410db060a91c18ae1c5e (table default.Table_f8b17194060c4627a7770e45799e082b [id=1c9da7e7dc974945b2b5ccf304d1c1a8])
I20251214 20:52:36.140690 2786 leader_election.cc:290] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:36.141039 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f1eb4c3aa654480caaa69dbd5047365c" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:36.141213 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e185d107619414a8bcc15ad496547dc" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:36.141418 1051 raft_consensus.cc:3060] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.141408 775 raft_consensus.cc:3060] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.144058 1051 raft_consensus.cc:2468] T 1e185d107619414a8bcc15ad496547dc P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.144115 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d5920fdba794e5f864c595e7be278ab" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.144243 1052 raft_consensus.cc:2468] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:36.145812 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57a6232564194cbba12182f3a29c2ebb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.145972 1052 raft_consensus.cc:2468] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:36.146075 775 raft_consensus.cc:2468] T f1eb4c3aa654480caaa69dbd5047365c P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.146680 848 leader_election.cc:304] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:36.146934 2946 tablet_loader.cc:96] loaded metadata for tablet 8a818ca230564b93ac165d43ff22deb1 (table default.Table_b289a25bf8174d9199c90212ce70271d [id=0a8ffa32e9ce4e3a8f9fa2afa4de69bb])
I20251214 20:52:36.147591 3221 raft_consensus.cc:2804] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.147729 3221 raft_consensus.cc:697] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.147964 3221 consensus_queue.cc:237] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.148038 2946 tablet_loader.cc:96] loaded metadata for tablet 8a977db125314550891491dcc53135cd (table default.Table_17954423e5674f0eb5a9b53bc2e75913 [id=c81e9d8bbee64045af7fce5cc8f64779])
I20251214 20:52:36.148175 2890 consensus_queue.cc:1048] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.148510 2890 consensus_queue.cc:1048] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.148837 2946 tablet_loader.cc:96] loaded metadata for tablet 8c07b455b12145e28ecff35f2732b840 (table default.Table_dd6d020171b24861a33a63f13b128050 [id=aa336c07501b4a99afc12338d7556623])
I20251214 20:52:36.149127 851 leader_election.cc:304] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.149461 2946 tablet_loader.cc:96] loaded metadata for tablet 8d72f55d2d2440d993aa205d96c356f1 (table default.Table_407ddd842d284ecd87a391f1b54a2b5a [id=66d124c2a8dd4cc188a832eff1dbe9ed])
I20251214 20:52:36.149479 851 leader_election.cc:304] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.150063 851 leader_election.cc:304] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.150211 2946 tablet_loader.cc:96] loaded metadata for tablet 8db4364b5023438bb43538855007c95e (table default.Table_80d7626996ca40979563d484ae097f08 [id=694ef81722d948638bbec8847c69d75a])
I20251214 20:52:36.150635 2777 consensus_queue.cc:1048] T 76472ce9005b4bd2bbc51278816e8ae0 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.150964 2777 raft_consensus.cc:2804] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.151048 2777 raft_consensus.cc:697] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.151269 2777 consensus_queue.cc:237] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.151506 3111 raft_consensus.cc:2804] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.151556 3111 raft_consensus.cc:493] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.151605 3111 raft_consensus.cc:3060] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.151598 2946 tablet_loader.cc:96] loaded metadata for tablet 8deabfdbd0674b2884ee5f82fa5e332f (table default.Table_3bc5f340978c453190d06f9e7a87f65c [id=87835c51994146ffa7f67ece6f50ff40])
I20251214 20:52:36.152817 2946 tablet_loader.cc:96] loaded metadata for tablet 8fcdb2ca52b14cf193d9ae87d0a50c56 (table default.Table_a1cc2f56735544ad8ae3425ad4d00fdb [id=bec715d329a74c7a8092f8e9c7f9c341])
I20251214 20:52:36.153213 2946 tablet_loader.cc:96] loaded metadata for tablet 8fcdd6bafa5c4aa6b59668cde523d010 (table default.Table_fe0d773656e24c0e9d6e2e36caeaa9f8 [id=ed335fb680bc441184e1682200ee68be])
I20251214 20:52:36.153566 2946 tablet_loader.cc:96] loaded metadata for tablet 91798ce3e88e4f82b59c52222dd83aef (table default.Table_8cd3b56154ec4d9fac4ed571a2287cfa [id=39df4da3b39545d3ba2c9345618ae532])
I20251214 20:52:36.154145 3111 raft_consensus.cc:515] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.154399 2946 tablet_loader.cc:96] loaded metadata for tablet 95c84b7c32f34490a2869d036345e4c4 (table default.Table_e4ed29947e6544b9a23370d584496d4a [id=daf3f737290e4b789b9f81ed0f0fa375])
I20251214 20:52:36.154667 3111 leader_election.cc:290] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.154788 2946 tablet_loader.cc:96] loaded metadata for tablet 96157aae81624a4ba9fcf8c3f142911f (table default.Table_adb418bf04534b868c044988fb97e906 [id=d7e403a944e94d0d8c9d6ba115723ea4])
I20251214 20:52:36.154973 3111 raft_consensus.cc:2804] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.155032 3111 raft_consensus.cc:493] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.155081 3111 raft_consensus.cc:3060] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.155160 2946 tablet_loader.cc:96] loaded metadata for tablet 96a2b444608e42d687eeba2dc0ebb757 (table default.Table_90e94bab31e54a9cb8b567622aa4e37d [id=7c3d3542270e449ca954d068bece3491])
I20251214 20:52:36.156647 1203 tablet_bootstrap.cc:492] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.156821 1203 ts_tablet_manager.cc:1403] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.002s
I20251214 20:52:36.159171 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d5920fdba794e5f864c595e7be278ab" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:36.161098 848 leader_election.cc:343] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.161161 1203 raft_consensus.cc:359] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.161525 1202 tablet_bootstrap.cc:492] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:36.161564 1203 raft_consensus.cc:385] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.162117 1203 raft_consensus.cc:740] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.162513 1203 consensus_queue.cc:260] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.162734 3111 raft_consensus.cc:515] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.172643 2946 tablet_loader.cc:96] loaded metadata for tablet 99663ff0fe094849acefe03db8d43b50 (table default.Table_581df19239f343f89fbe7e5c17d6d85c [id=2c6c7db9251948fe840e9fc0cc2076db])
I20251214 20:52:36.173305 2946 tablet_loader.cc:96] loaded metadata for tablet 9bd89709abbc49809a2cdb6d0c29fdb8 (table default.Table_c3aa8e3c31cb41dc8281962e7b5c46e1 [id=687eb544b4114bad8d2deb76388e2022])
I20251214 20:52:36.173641 2946 tablet_loader.cc:96] loaded metadata for tablet 9bf9e65040af4fbeb701619fe27edfb6 (table default.Table_91b2da75e7104e0f95ed470c749f843f [id=70aba52f026e439f8bf0b12373148fca])
I20251214 20:52:36.174263 2946 tablet_loader.cc:96] loaded metadata for tablet 9cc05b7f5c114604afa025338b6fa617 (table default.Table_b3dbc2e42cc64d9d96c48ecc1b474480 [id=e78b07fcb7a341fdbd0c244409c3c445])
I20251214 20:52:36.174611 2946 tablet_loader.cc:96] loaded metadata for tablet 9cfd9b2f9756433e9e52643417c4cd45 (table default.Table_ba49861ed76e45f08158052d3e3fec58 [id=69175aeba7894328af1e0fe73b55025d])
I20251214 20:52:36.175012 2946 tablet_loader.cc:96] loaded metadata for tablet 9d56346fa26247f38ef716a9a8afac1a (table default.Table_debb0f541d1b4ca3bc00c5e2c81c8428 [id=3b70fc138995441896f4ec7341fb69d4])
I20251214 20:52:36.175663 2946 tablet_loader.cc:96] loaded metadata for tablet 9da5e98c6f4843449886176e9fcfa8e1 (table default.Table_55f303d33ec84a269232fc1ba8f2d182 [id=5ded9fe825ae4308b5bd4b21c16634c8])
I20251214 20:52:36.176299 2946 tablet_loader.cc:96] loaded metadata for tablet 9f04336d2c84439b9003df19d3541273 (table default.Table_7041fb3e6d4040f08fd9a554a30e315d [id=b6e9928c0c2845fdb544b4b449aeb113])
I20251214 20:52:36.177306 2946 tablet_loader.cc:96] loaded metadata for tablet a07d94d5a431488489ee9fbcc113a9d7 (table default.Table_493f60b9d8024509856ed934d379f829 [id=a40a329d98674364aa3794d3a07fdeb4])
I20251214 20:52:36.178148 2946 tablet_loader.cc:96] loaded metadata for tablet a12e8053c65343758d518ee840dbaee8 (table default.Table_cb922bf98617480aa9363c3f08d918f3 [id=e815640c7ece4361800a22bc749915e9])
I20251214 20:52:36.178691 2946 tablet_loader.cc:96] loaded metadata for tablet a34d3b8893224b5f94014c2da3d5ae65 (table default.Table_98b0ed640a93417ba49359781348b105 [id=5addd19777df4b968708a41c59a19651])
I20251214 20:52:36.171254 1203 ts_tablet_manager.cc:1434] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.014s user 0.002s sys 0.002s
W20251214 20:52:36.180274 713 leader_election.cc:343] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 98b7cac3930a4a9eb16ac413c8f1e092
I20251214 20:52:36.179767 1203 tablet_bootstrap.cc:492] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.181512 1206 tablet_bootstrap.cc:492] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:36.181638 1206 ts_tablet_manager.cc:1403] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.001s
I20251214 20:52:36.182375 1206 raft_consensus.cc:359] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.182552 1206 raft_consensus.cc:385] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.182600 1206 raft_consensus.cc:740] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.182816 1206 consensus_queue.cc:260] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.186389 1206 ts_tablet_manager.cc:1434] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:36.186693 1206 tablet_bootstrap.cc:492] T dfe50a6219134431b0a025317932609b P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:36.188652 1206 tablet_bootstrap.cc:654] T dfe50a6219134431b0a025317932609b P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.189589 2991 raft_consensus.cc:493] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.189706 2991 raft_consensus.cc:515] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.190243 2991 leader_election.cc:290] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.191259 1203 tablet_bootstrap.cc:654] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.194128 2946 tablet_loader.cc:96] loaded metadata for tablet a41ed28272ad4e27bd2203dc4d3ad113 (table default.Table_e552103bd07e4f68b25576d6044d8861 [id=80a316658e6d494387628a03406f6480])
I20251214 20:52:36.195664 2946 tablet_loader.cc:96] loaded metadata for tablet a4643effa71d4d18850913a59ae1c75d (table default.Table_679d53b1472341ef8612ca53ac1ebda2 [id=91f3b4dd176348a5aa38c6bdc417918f])
W20251214 20:52:36.196578 2288 consensus_peers.cc:597] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad -> Peer c27224391721467a8f46e467a980b37a (127.0.104.62:42865): Couldn't send request to peer c27224391721467a8f46e467a980b37a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42865: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251214 20:52:36.196822 3119 raft_consensus.cc:493] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.196976 3119 raft_consensus.cc:515] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.196630 2946 tablet_loader.cc:96] loaded metadata for tablet a4bcac304fd14e20b0607c23d1569a31 (table default.Table_d8bb40c2324c4c0eb7e52aa96e920813 [id=3b04d86591b14695a334b68b0572f371])
I20251214 20:52:36.197535 2946 tablet_loader.cc:96] loaded metadata for tablet a5d28f09ad174936b47debace0599600 (table default.Table_93730665b1ab4686a0d3e54740fbddc8 [id=0b5581be22d345b3a0fd22c3dce2f207])
I20251214 20:52:36.197600 3119 leader_election.cc:290] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.198050 2991 raft_consensus.cc:493] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.198181 2991 raft_consensus.cc:515] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.198246 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7242be189ca54c9182181e33d6743e6b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.198698 2991 leader_election.cc:290] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:36.198946 2946 tablet_loader.cc:96] loaded metadata for tablet a6517a7974cd430e95d976fa3f99e465 (table default.Table_d9babf534d4b4411aa03906984198232 [id=2e2d38b971be42a1befe0d6995030970])
I20251214 20:52:36.199756 2946 tablet_loader.cc:96] loaded metadata for tablet a6972d034dd54bd880dc09a3fb983d0c (table default.Table_4fdc1b366f04459b8b083811e9740ae1 [id=4ba4d4bb0a2d48b4aa7b956cafdb26a7])
W20251214 20:52:36.199738 989 leader_election.cc:343] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.164237 1202 tablet_bootstrap.cc:654] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:36.186667 848 consensus_peers.cc:597] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:36.205304 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7242be189ca54c9182181e33d6743e6b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.207101 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e621f29c01448399d22615284fccd4" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.210647 3111 leader_election.cc:290] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.211472 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57a6232564194cbba12182f3a29c2ebb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:36.211766 991 leader_election.cc:343] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.211894 991 leader_election.cc:304] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.212350 3119 raft_consensus.cc:2749] T 7242be189ca54c9182181e33d6743e6b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:36.212812 848 leader_election.cc:343] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.212894 2946 tablet_loader.cc:96] loaded metadata for tablet a78664a3046d4c61aa20ba86ed9e486e (table default.Table_eaf150707b2442f1ae6f94ac564c6b87 [id=67577dd83c4045c992f1c574bd71edfe])
I20251214 20:52:36.213809 2946 tablet_loader.cc:96] loaded metadata for tablet a9c383e343544c83ada89bea6ab20e94 (table default.Table_379fdcc125644411b3d7172ec3d66698 [id=dd60146ee388428c867677cb19d02589])
I20251214 20:52:36.214219 2946 tablet_loader.cc:96] loaded metadata for tablet aa5e66f9d8a8410da1f349788a0a8ad8 (table default.Table_5b1cf9007e0b4c3bae65a57ec193d8fa [id=cffdf6b3ab8b4d109c4552b0472fe6e9])
I20251214 20:52:36.214622 2946 tablet_loader.cc:96] loaded metadata for tablet aaf060ab2f71429a85dda19669e69677 (table default.Table_9321256005b04155b2f5a451ce5065fe [id=c5a7a8665d6647bdb19ed6ef27907fee])
I20251214 20:52:36.215662 2946 tablet_loader.cc:96] loaded metadata for tablet ad99467157df4c37bb9b518244a5dbd8 (table default.Table_e125108df3c94234bedbdf467db81f28 [id=a47338e44a9e4642baa8407af70be735])
I20251214 20:52:36.216073 2946 tablet_loader.cc:96] loaded metadata for tablet ae54e7a5104e4cd588a254551fc52477 (table default.Table_68c79df4e9e0446f81c8500bcfa423a6 [id=7e9bbbe22677488d9509dc201b2d7234])
I20251214 20:52:36.216754 2946 tablet_loader.cc:96] loaded metadata for tablet afd51f330eb649b998a22a343d993b30 (table default.Table_9151bdce51684c39901786aef1c63ec5 [id=f4157d7b9fc848a8b7fa08742b456167])
I20251214 20:52:36.219103 2946 tablet_loader.cc:96] loaded metadata for tablet b51d8e5dc7e04fdb9c9d3180c1fe9bdb (table default.Table_09270706a48c42fea27376b3ef3a0746 [id=3f15c09d26b04d519d92594446cca818])
I20251214 20:52:36.220604 2946 tablet_loader.cc:96] loaded metadata for tablet b67190c21429415081c867332b080f73 (table default.Table_183ef00a0ed14492bb36f72ed1c73aa2 [id=a61d5b8118774bb2b806d755ab98da86])
I20251214 20:52:36.221549 1203 tablet_bootstrap.cc:492] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.221803 1203 ts_tablet_manager.cc:1403] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.004s
I20251214 20:52:36.222402 2946 tablet_loader.cc:96] loaded metadata for tablet b8727f302d35471eb2eac1e77b8d966b (table default.Table_01b1f692968d4dbe8d92746b6017a8b3 [id=eefab419a1984e92997ff7429a6dd368])
I20251214 20:52:36.222826 2946 tablet_loader.cc:96] loaded metadata for tablet b87b0871c2454971a15dd5643c155694 (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749])
I20251214 20:52:36.222715 1203 raft_consensus.cc:359] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.222898 1203 raft_consensus.cc:385] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.223058 1203 raft_consensus.cc:740] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.223367 1203 consensus_queue.cc:260] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.223624 2946 tablet_loader.cc:96] loaded metadata for tablet ba322cb6ee084bc39788ddd439f5f15d (table default.Table_196d0f488e0b42aabd518511a4289dc3 [id=2f57c28c9c624a6ca1986c8831b85169])
W20251214 20:52:36.223706 713 leader_election.cc:343] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: ae948ffc81764692b92045f5befe0184
I20251214 20:52:36.224088 1203 ts_tablet_manager.cc:1434] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:36.224426 1203 tablet_bootstrap.cc:492] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:36.224933 713 consensus_peers.cc:597] T bc73d590531c409e9f60670435dc8223 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc73d590531c409e9f60670435dc8223. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:36.225219 2946 tablet_loader.cc:96] loaded metadata for tablet ba548574d8784701974e611d7a9ad9b1 (table default.Table_0c936131512c4d99b3999d74fabf18d3 [id=1c5148c58b1c4fe39bac0697004ca484])
I20251214 20:52:36.225665 2946 tablet_loader.cc:96] loaded metadata for tablet ba7952290e2c4746802cffeb14971da0 (table default.Table_d677f51980c54afd9712bf047e17f49e [id=be511c2e8a9a4059bec0bf6ac1b68741])
I20251214 20:52:36.226207 1203 tablet_bootstrap.cc:654] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.227191 2946 tablet_loader.cc:96] loaded metadata for tablet bd17ca86d3a243d78d1f782c56f51a48 (table default.Table_70e73cbd7f9141f3ab482647b6118d52 [id=91a01b7a64464481b8ebe6331c0746da])
I20251214 20:52:36.228335 2946 tablet_loader.cc:96] loaded metadata for tablet bea3e9d7334345fa88ededf1be17a434 (table default.Table_27579b7155b140b3bdda5014c1eb57a8 [id=e24a6c9e5c9049edba5e42e22d22a831])
I20251214 20:52:36.232184 2946 tablet_loader.cc:96] loaded metadata for tablet c5478cba0bf24f8c969fd1b0998b9db3 (table default.Table_0f409000b9514206a08757040e81a6a9 [id=5692c4a553044543ab275cff0f7a0566])
I20251214 20:52:36.232640 2946 tablet_loader.cc:96] loaded metadata for tablet c6a8863ace4541aea7463af24db27142 (table default.Table_88881a27865d4b0aa6db6a2b736f0c17 [id=e0d1675617c74aa8a8afd81bd90f9639])
I20251214 20:52:36.233101 2946 tablet_loader.cc:96] loaded metadata for tablet c6eab2148fbf412483f1a4723123f41f (table default.Table_7219830d8fb040debc2388f49091b802 [id=40d3f605109b43828c497df2bf4eef65])
I20251214 20:52:36.233513 2946 tablet_loader.cc:96] loaded metadata for tablet c7248a2e8355425fa051a197160dbdbb (table default.Table_4bb4b496a62d4c89b7b4e783023dd5cd [id=144f1eb0434a44f49e52b76ba687525c])
I20251214 20:52:36.233927 2946 tablet_loader.cc:96] loaded metadata for tablet c75e6701210e4d688a96678bf042860a (table default.Table_040e46c43e8446f48a60e34f0892bf57 [id=2b6d1f218393414d868d37d390b35b58])
I20251214 20:52:36.234356 2946 tablet_loader.cc:96] loaded metadata for tablet c7885f21a369461c8c80ff18d369e2c6 (table default.Table_7fbc660486544d9fb711dcdca0660237 [id=a20045f442604c45a67b9b06f1eb4610])
I20251214 20:52:36.234761 2946 tablet_loader.cc:96] loaded metadata for tablet c86bb092060b4019ac0f557bfd3f14f5 (table default.Table_09f08f5ffa814302a74b268170036a24 [id=71cefadd096649a8aca97ece15ff1bb9])
I20251214 20:52:36.240758 2946 tablet_loader.cc:96] loaded metadata for tablet c8e43456af5449939107def570689b7c (table default.Table_cb1220aeec334a3c8aa15d0049c58d75 [id=3e833b0eaf3a4f519a1c1cfebaa0e969])
I20251214 20:52:36.242857 2946 tablet_loader.cc:96] loaded metadata for tablet cab1bed6016243bda0d9bd7f512884c8 (table default.Table_c3bedc14bbf14c5988f9c60ceacf9d95 [id=58ace009e5fc4eab93d8babc6330f58e])
I20251214 20:52:36.243968 2946 tablet_loader.cc:96] loaded metadata for tablet cc8d725e6ac440899fc1cde2f7b546aa (table default.Table_1052ac11eee9483aa13db090acff5b7a [id=978e62482f114f2b99e9707aaa9a5e32])
I20251214 20:52:36.244390 2946 tablet_loader.cc:96] loaded metadata for tablet ccf2bd74820b4e9e843265343293cad1 (table default.Table_ac4df504ee284227bfd4ba2a23eaff65 [id=5f69f1d7797b4fff9030914e710a8fb8])
I20251214 20:52:36.245045 2946 tablet_loader.cc:96] loaded metadata for tablet cdd2a91f9d4c44548966f8e68a75d8de (table default.Table_1aadd2d8580e419fb5a3767b09353c25 [id=e4d01eb0c006487bb2f14fb3e14dd18f])
I20251214 20:52:36.245952 2946 tablet_loader.cc:96] loaded metadata for tablet cf04f91e79754569a51d468ab576b0dc (table default.Table_7abff44151df46dd935898812837e8b8 [id=98edd613a48b433c9896ff066362554d])
W20251214 20:52:36.246600 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 96528.8us
W20251214 20:52:36.246935 851 leader_election.cc:343] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: f1eb4c3aa654480caaa69dbd5047365c
W20251214 20:52:36.247422 851 leader_election.cc:343] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: f1eb4c3aa654480caaa69dbd5047365c
I20251214 20:52:36.248059 2946 tablet_loader.cc:96] loaded metadata for tablet d2688515bd1441fbad7516f8343ca775 (table default.Table_f1e9c435093240e4b1c1eb95635c8e66 [id=03997e2eab154ef18b2e3774b5984764])
W20251214 20:52:36.248976 712 leader_election.cc:343] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 98b7cac3930a4a9eb16ac413c8f1e092
I20251214 20:52:36.249145 712 leader_election.cc:304] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.249213 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2d5920fdba794e5f864c595e7be278ab" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:36.249466 1055 raft_consensus.cc:3060] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.249692 2991 raft_consensus.cc:2749] T 98b7cac3930a4a9eb16ac413c8f1e092 P 074399864fb2470fa9e2ee87c1fe2212 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251214 20:52:36.252838 1055 raft_consensus.cc:2468] T 2d5920fdba794e5f864c595e7be278ab P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.253616 2946 tablet_loader.cc:96] loaded metadata for tablet d39e245bc5a04c1186d0973645aa1b54 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7])
I20251214 20:52:36.254245 2946 tablet_loader.cc:96] loaded metadata for tablet d424e3387698434f92e84eb1d43f5949 (table default.Table_9098a11909c844dca6d4e80a284aa2e8 [id=1597e24e3fff4007ab66b44505c960d8])
I20251214 20:52:36.254670 2946 tablet_loader.cc:96] loaded metadata for tablet d48c6789e5724e989900d3351ecde3ab (table default.Table_b6011f3e00584819944d165d653d06ac [id=103545908ce040858f00aed970b27479])
I20251214 20:52:36.254544 2786 consensus_queue.cc:1048] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.255056 2946 tablet_loader.cc:96] loaded metadata for tablet d5e0b89d02c04625b8e31910d60d5d40 (table default.Table_b72c21c5b00d430ea93eff8bf8bcb228 [id=4d54f97a2cde4143a185eb4af35bb07d])
I20251214 20:52:36.256093 3141 raft_consensus.cc:493] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.256470 2946 tablet_loader.cc:96] loaded metadata for tablet d9f198ef4dfa4f5791144b9a24fe693f (table default.Table_1a531b1c0ece4bfb9b015d3285d4c3b3 [id=fc592ea105f242a299337c9cbe438778])
I20251214 20:52:36.256322 3141 raft_consensus.cc:515] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.257593 3141 leader_election.cc:290] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.257758 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf9e65040af4fbeb701619fe27edfb6" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:36.258617 848 leader_election.cc:343] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:36.259855 848 consensus_peers.cc:597] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:36.259651 2946 tablet_loader.cc:96] loaded metadata for tablet db0ad1aa4193467db74cd707e7084b71 (table default.Table_5c7eb24742b84515af287c257637e51b [id=8acc06886693413682b246f25970fa5b])
I20251214 20:52:36.261157 2946 tablet_loader.cc:96] loaded metadata for tablet dcecde5ed91c48689bed1d36bfd5725e (table default.Table_eba09dc20b824904b171bc39be41ea65 [id=a4ef8163c0a741f290495ef9015ae6cb])
I20251214 20:52:36.261646 2946 tablet_loader.cc:96] loaded metadata for tablet ddd625b1cf4a4af191ad4e9a85f7b976 (table default.Table_58bb2f3098084d73bafbd51cde268a21 [id=1e84b30c44fc49ba8a8b35c7e3252761])
I20251214 20:52:36.262128 2946 tablet_loader.cc:96] loaded metadata for tablet de9c830cf9c74cff88cb6f42cbb5f7cd (table default.Table_f9499a2114f44353aaf65b35ed56f119 [id=7dd81c38dde94205a84d6f7841d2bb73])
I20251214 20:52:36.262796 2946 tablet_loader.cc:96] loaded metadata for tablet e0d0e8f35bf44c9a8797b7507dcce3b9 (table default.Table_4649d8b3e43c42a492723fadd4217e4e [id=75cebf8d0aba4aa48c1be51b0abd6b38])
I20251214 20:52:36.263477 2946 tablet_loader.cc:96] loaded metadata for tablet e1dbaa39efa04086a22cc3908ed691d4 (table default.Table_8f897cb4591a4d5fadbf2b1b7ee7408d [id=4390cb960d564bb5bc7e0cefa5bf3004])
I20251214 20:52:36.263841 2946 tablet_loader.cc:96] loaded metadata for tablet e3696c623bb0407a8fa4f7ee88409b04 (table default.Table_a5e4f40e603f4fd0a3d204f9c0298023 [id=36a5b0a44e8548e0a3c3f28975c4337d])
I20251214 20:52:36.264714 2946 tablet_loader.cc:96] loaded metadata for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_5de6830aaec645f2992e4cfaceb20c27 [id=9553b46856c94003a293135c814a7df7])
I20251214 20:52:36.265554 2946 tablet_loader.cc:96] loaded metadata for tablet e8c02a2e4a2640b1b3a69f25a708684b (table default.Table_a519bf6d1ab8419784be3772e899869d [id=5cbe2438b4294442a9778bf1f641b60e])
I20251214 20:52:36.266446 2946 tablet_loader.cc:96] loaded metadata for tablet e8c4693bad7a485ea769ab075c536b35 (table default.Table_a54f22962823421ea5ab3033e0d9add0 [id=5db835c4244847e4969597e99df50099])
I20251214 20:52:36.266422 2991 raft_consensus.cc:493] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.266623 2991 raft_consensus.cc:515] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.267511 2991 leader_election.cc:290] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:36.267727 2946 tablet_loader.cc:96] loaded metadata for tablet eb3ff5864734456090879ec7f06de288 (table default.Table_87794c3eaa7c4545acddac9ad4b4bc93 [id=59e0f0da4a1447a6957080f16c820e19])
I20251214 20:52:36.268728 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7aa508b11944aa8e268741c1998a9e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.269842 1206 tablet_bootstrap.cc:492] T dfe50a6219134431b0a025317932609b P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:36.269917 2946 tablet_loader.cc:96] loaded metadata for tablet ed37194008044d2d84421b387f66dbba (table default.Table_691250c9129d453585ad50b32a1e340e [id=edcea358cf1c4046a473b27b94ba56ac])
I20251214 20:52:36.270114 1206 ts_tablet_manager.cc:1403] T dfe50a6219134431b0a025317932609b P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.083s user 0.001s sys 0.008s
I20251214 20:52:36.270521 2946 tablet_loader.cc:96] loaded metadata for tablet edcf8857116d4394b66f5fa7fce343a5 (table default.Table_fc1d7cb6f8544f6285ec8792deb1da74 [id=8ce51bbd64ff4e1789fade77a6b0c13f])
I20251214 20:52:36.270996 1206 raft_consensus.cc:359] T dfe50a6219134431b0a025317932609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.271291 1206 raft_consensus.cc:385] T dfe50a6219134431b0a025317932609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.271412 1206 raft_consensus.cc:740] T dfe50a6219134431b0a025317932609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.271514 2946 tablet_loader.cc:96] loaded metadata for tablet f11834cabdad445b8e53cd237d146282 (table default.Table_398f36b3acb440d88213e38f63740ca5 [id=954b3b4acdd94043b2608438d00483e7])
I20251214 20:52:36.271723 1206 consensus_queue.cc:260] T dfe50a6219134431b0a025317932609b P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.273415 2946 tablet_loader.cc:96] loaded metadata for tablet f5f6cbcb80e4428898148180080273c8 (table default.Table_94c723f6fee24f40a5787f7e5e92d03f [id=223e7226cfa94865a38476e84504a946])
I20251214 20:52:36.273831 2946 tablet_loader.cc:96] loaded metadata for tablet f6b7fe43fa104199be5363ca53035121 (table default.Table_1225d07e2e7a4aa1b60557b64de1f35e [id=07f7bb3503114ba3b0e1fe404826d582])
I20251214 20:52:36.274180 2946 tablet_loader.cc:96] loaded metadata for tablet f6d51d07f7e9450a844be9afddb60d03 (table default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3])
I20251214 20:52:36.275216 2946 tablet_loader.cc:96] loaded metadata for tablet f891db00595b49ca9e6ae64721e4d744 (table default.Table_c404f3a3b71f4899a233f864e78a7d13 [id=203681807a4b40b0a1f69387cfecff38])
I20251214 20:52:36.275648 2946 tablet_loader.cc:96] loaded metadata for tablet f8df264ec35c4a9d8a23f234d8c56bc6 (table default.Table_42e9fda7b7d14640a5804197bfb41683 [id=3bd3b6b8ee1f4d7abc6f888fc9d9004a])
I20251214 20:52:36.276072 2946 tablet_loader.cc:96] loaded metadata for tablet f8e98027ac3f4f36aded3aa5b47e6cad (table default.Table_5c7e3a65102c45a1a23d29b95427fae5 [id=db0d9e33bf054daabddadd1248243609])
I20251214 20:52:36.276474 2946 tablet_loader.cc:96] loaded metadata for tablet f900ebe92efa476cac232131a818d449 (table default.Table_5de120dfaab44b84868db0d1f79e2cee [id=1158f8a25d2b4696a5f5ed2c3a426862])
I20251214 20:52:36.276883 2946 tablet_loader.cc:96] loaded metadata for tablet f945814952b14131ba4e0c67a403a680 (table default.Table_75bf61ed45674e18a330a910511df0fc [id=70cb284b0267483690f5e826d6802bb4])
I20251214 20:52:36.277526 2946 tablet_loader.cc:96] loaded metadata for tablet fa141b2570514c6aacb2850e4bfd1869 (table default.Table_47baf32513e6487fbd4ffde9bcba97ca [id=5ea1e541e5454f1ca96338573fda5705])
I20251214 20:52:36.278178 2946 tablet_loader.cc:96] loaded metadata for tablet fd3a49d6acae40d099cdd99981ce9d46 (table default.Table_73af42b244d14d33889b754b78a9fd11 [id=6a3c23042b304db6b72d9d2d112884f7])
W20251214 20:52:36.281893 2946 catalog_manager.cc:1486] Time spent T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Loading table and tablet metadata into memory: real 2.152s user 0.304s sys 0.083s
I20251214 20:52:36.282054 2946 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251214 20:52:36.291978 1206 ts_tablet_manager.cc:1434] T dfe50a6219134431b0a025317932609b P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20251214 20:52:36.297750 2946 catalog_manager.cc:1269] Loaded cluster ID: c3f04138608d4adc935bb66e1a72f2bb
I20251214 20:52:36.298013 2946 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251214 20:52:36.302671 775 raft_consensus.cc:3060] T dfe50a6219134431b0a025317932609b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:36.306488 713 consensus_peers.cc:597] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 088c72a6f5014b6fbc6ee060b6b7a590. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:36.306720 2991 raft_consensus.cc:493] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.306856 2991 raft_consensus.cc:515] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.307672 2991 leader_election.cc:290] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:36.309135 713 leader_election.cc:343] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 9b1a937150ce4ecfa210c8465d682496
I20251214 20:52:36.309785 3111 raft_consensus.cc:493] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.309970 3111 raft_consensus.cc:515] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.310773 3111 leader_election.cc:290] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:36.311765 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eff8a23d48c43e2bdbfb34f4bab6edf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.314298 2397 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 8 candidate_status { last_received { term: 7 index: 956 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
I20251214 20:52:36.310395 2387 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c48b8eaf30c4e2caa94a51d087406d4" candidate_term: 8 candidate_status { last_received { term: 7 index: 956 } } ignore_live_leader: false dest_uuid: "e6713a20a82b434f8a8bc122113e26ad" is_pre_election: true
W20251214 20:52:36.315155 2397 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41550 (request call id 2) took 980 ms (client timeout 683 ms). Trace:
W20251214 20:52:36.315267 848 leader_election.cc:343] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:36.315280 2397 rpcz_store.cc:269] 1214 20:52:35.334385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:35.334450 (+ 65us) service_pool.cc:225] Handling call
1214 20:52:36.315070 (+980620us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 20:52:36.314780 2326 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:41562 (request call id 29) took 2163 ms. Trace:
I20251214 20:52:36.316414 2326 rpcz_store.cc:276] 1214 20:52:34.151660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:34.151790 (+ 130us) service_pool.cc:225] Handling call
1214 20:52:36.314692 (+2162902us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 20:52:36.316553 1206 tablet_bootstrap.cc:492] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:36.317488 1297 leader_election.cc:304] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c48b8eaf30c4e2caa94a51d087406d4; no voters: c27224391721467a8f46e467a980b37a, e6713a20a82b434f8a8bc122113e26ad
I20251214 20:52:36.318511 3250 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7c48b8eaf30c4e2caa94a51d087406d4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251214 20:52:36.314769 2946 catalog_manager.cc:1514] Loading token signing keys...
I20251214 20:52:36.324972 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57a6232564194cbba12182f3a29c2ebb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:36.325194 1055 raft_consensus.cc:3060] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.326711 1202 tablet_bootstrap.cc:492] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:36.327806 1202 ts_tablet_manager.cc:1403] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.166s user 0.002s sys 0.008s
I20251214 20:52:36.328318 1055 raft_consensus.cc:2468] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.329836 1202 raft_consensus.cc:359] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.330345 1202 raft_consensus.cc:385] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.330550 1202 raft_consensus.cc:740] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.330994 1202 consensus_queue.cc:260] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.331537 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae948ffc81764692b92045f5befe0184" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.331712 908 raft_consensus.cc:2468] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:36.331805 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1d7aa508b11944aa8e268741c1998a9e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.331902 1202 ts_tablet_manager.cc:1434] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:36.332013 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "79e621f29c01448399d22615284fccd4" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.333057 1202 tablet_bootstrap.cc:492] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:36.334623 1202 tablet_bootstrap.cc:654] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.336916 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf9e65040af4fbeb701619fe27edfb6" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.337154 1056 raft_consensus.cc:2468] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:36.339219 3119 raft_consensus.cc:493] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.339339 712 leader_election.cc:304] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:36.339367 3119 raft_consensus.cc:515] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.339908 3119 leader_election.cc:290] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.339905 712 leader_election.cc:304] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.340438 3098 raft_consensus.cc:2804] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.340554 3098 raft_consensus.cc:493] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.340651 3098 raft_consensus.cc:3060] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.340935 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "185049f1ef43499aa1d15d99549255f7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:36.341660 989 leader_election.cc:343] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.342411 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eff8a23d48c43e2bdbfb34f4bab6edf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.342564 1052 raft_consensus.cc:2468] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:36.343024 851 leader_election.cc:304] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.343307 3039 raft_consensus.cc:2749] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.343335 2786 raft_consensus.cc:2804] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.343464 2786 raft_consensus.cc:697] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.344094 2786 consensus_queue.cc:237] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.344607 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "185049f1ef43499aa1d15d99549255f7" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.344648 3098 raft_consensus.cc:515] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.345453 3098 leader_election.cc:290] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:36.345929 991 leader_election.cc:343] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.346047 991 leader_election.cc:304] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.346259 3119 raft_consensus.cc:2749] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:36.347563 713 leader_election.cc:343] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: ae948ffc81764692b92045f5befe0184
I20251214 20:52:36.351172 2777 consensus_queue.cc:1048] T 4e4dc66681ef48479ae3cb95b091564a P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.353554 2777 consensus_queue.cc:1048] T a07d94d5a431488489ee9fbcc113a9d7 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.354593 851 leader_election.cc:304] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.359949 2889 raft_consensus.cc:2804] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.360349 2889 raft_consensus.cc:697] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.360685 2889 consensus_queue.cc:237] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.367877 1206 tablet_bootstrap.cc:654] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.368619 1203 tablet_bootstrap.cc:492] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.368813 1203 ts_tablet_manager.cc:1403] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.144s user 0.043s sys 0.054s
I20251214 20:52:36.369623 1203 raft_consensus.cc:359] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.369848 1203 raft_consensus.cc:385] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.369910 1203 raft_consensus.cc:740] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.370118 1203 consensus_queue.cc:260] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.370517 1203 ts_tablet_manager.cc:1434] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:36.370821 1203 tablet_bootstrap.cc:492] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.367070 851 leader_election.cc:304] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.372550 2777 raft_consensus.cc:2804] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.372650 2777 raft_consensus.cc:493] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.372733 2777 raft_consensus.cc:3060] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.373854 3098 raft_consensus.cc:493] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.373906 2946 catalog_manager.cc:6038] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Loaded TSK: 0
I20251214 20:52:36.374051 3098 raft_consensus.cc:515] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:36.375003 3098 leader_election.cc:290] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:36.375552 2946 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251214 20:52:36.375475 2777 raft_consensus.cc:515] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.376091 2777 leader_election.cc:290] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.376597 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf9e65040af4fbeb701619fe27edfb6" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:36.377727 2777 consensus_queue.cc:1048] T 054cfd587d19409fbdf7f4848bcaf3f9 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.378414 2777 consensus_queue.cc:1048] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.378875 851 leader_election.cc:304] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
W20251214 20:52:36.379280 848 leader_election.cc:343] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.379359 2777 raft_consensus.cc:2804] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.379630 2777 raft_consensus.cc:493] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.379741 3111 consensus_queue.cc:1048] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.379851 2777 raft_consensus.cc:3060] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.383283 2777 raft_consensus.cc:515] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.384100 2777 leader_election.cc:290] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:36.385047 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eff8a23d48c43e2bdbfb34f4bab6edf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:36.385979 848 leader_election.cc:343] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.392602 2889 raft_consensus.cc:493] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.392889 2889 raft_consensus.cc:515] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.393648 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bf9e65040af4fbeb701619fe27edfb6" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:36.393851 1056 raft_consensus.cc:3060] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.394047 1203 tablet_bootstrap.cc:654] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:36.402446 713 leader_election.cc:343] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 25dcf491653147e7831217c1ec7a2e0c
I20251214 20:52:36.404106 1206 tablet_bootstrap.cc:492] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:36.404289 1206 ts_tablet_manager.cc:1403] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.000s
I20251214 20:52:36.405342 1206 raft_consensus.cc:359] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.405577 1206 raft_consensus.cc:385] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.405653 1206 raft_consensus.cc:740] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.405896 1206 consensus_queue.cc:260] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:36.406608 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 66348.3us
I20251214 20:52:36.406924 712 leader_election.cc:304] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.407224 775 raft_consensus.cc:3060] T 292a28b12df246a8bc855d81c530746e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.409129 1202 tablet_bootstrap.cc:492] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:36.409314 1202 ts_tablet_manager.cc:1403] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.001s
I20251214 20:52:36.410075 1202 raft_consensus.cc:359] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.410305 1202 raft_consensus.cc:385] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.410403 1202 raft_consensus.cc:740] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.396714 1056 raft_consensus.cc:2468] T 9bf9e65040af4fbeb701619fe27edfb6 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.410646 1202 consensus_queue.cc:260] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.411278 1202 ts_tablet_manager.cc:1434] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:36.411557 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1eff8a23d48c43e2bdbfb34f4bab6edf" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:36.411782 1056 raft_consensus.cc:3060] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.412016 1202 tablet_bootstrap.cc:492] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:36.413899 1202 tablet_bootstrap.cc:654] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.415071 1056 raft_consensus.cc:2468] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.430387 2325 catalog_manager.cc:5654] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:36.431288 2325 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 566c6e4761d5486d964e276d151e32b0 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 2 got 0
I20251214 20:52:36.432423 2889 leader_election.cc:290] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.432541 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d745583012b4465090d85c75470e0bc2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:36.433084 848 leader_election.cc:343] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.435364 851 leader_election.cc:304] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.435847 2786 raft_consensus.cc:2804] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.436003 2786 raft_consensus.cc:697] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.436314 2786 consensus_queue.cc:237] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.442278 3119 raft_consensus.cc:493] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.442492 3119 raft_consensus.cc:515] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.443305 3119 leader_election.cc:290] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.444077 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7b247b761c94c60ad3a468db371b30b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.444823 2321 master_service.cc:954] ReplaceTablet: received request to replace tablet c7248a2e8355425fa051a197160dbdbb from {username='slave'} at 127.0.0.1:41562
W20251214 20:52:36.445103 991 leader_election.cc:343] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.445598 1206 ts_tablet_manager.cc:1434] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20251214 20:52:36.446118 2320 master_service.cc:954] ReplaceTablet: received request to replace tablet 06ebe9f717bf43108748605cf4f961b7 from {username='slave'} at 127.0.0.1:41562
I20251214 20:52:36.449544 1206 tablet_bootstrap.cc:492] T 80098b7c24824d129b8f25daf2115460 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:36.450282 2991 raft_consensus.cc:2749] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:36.450438 712 leader_election.cc:343] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 9b1a937150ce4ecfa210c8465d682496
I20251214 20:52:36.450560 712 leader_election.cc:304] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.450796 2991 raft_consensus.cc:2749] T 9b1a937150ce4ecfa210c8465d682496 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:36.451387 2316 master_service.cc:954] ReplaceTablet: received request to replace tablet fa141b2570514c6aacb2850e4bfd1869 from {username='slave'} at 127.0.0.1:41562
I20251214 20:52:36.451640 851 leader_election.cc:304] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.452332 3221 raft_consensus.cc:2804] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.452447 3221 raft_consensus.cc:697] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.452713 3221 consensus_queue.cc:237] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.453109 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7b247b761c94c60ad3a468db371b30b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:36.454020 989 leader_election.cc:343] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.454166 989 leader_election.cc:304] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.454576 3119 raft_consensus.cc:2749] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.455227 3039 raft_consensus.cc:493] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.455384 3039 raft_consensus.cc:515] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.456034 3039 leader_election.cc:290] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:36.464929 713 leader_election.cc:343] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: d01db63fd5d2474a9defdce0b98fc29d
I20251214 20:52:36.466396 1206 tablet_bootstrap.cc:654] T 80098b7c24824d129b8f25daf2115460 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.475591 1203 tablet_bootstrap.cc:492] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.440428 2326 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_9ff099dcfac14636a9dcaa70207c86a6" } new_table_name: "default.Table_b77f9ca5f0f84a4db069e0a93db601e9" modify_external_catalogs: true
I20251214 20:52:36.443603 2328 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_f9e379a41675483dbc3099b79604781a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:36.494477 2328 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9e379a41675483dbc3099b79604781a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.443941 2330 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_c15b44a5f25a4a38aa2d18ff7a59e5c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:36.496734 2330 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c15b44a5f25a4a38aa2d18ff7a59e5c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.497538 1206 tablet_bootstrap.cc:492] T 80098b7c24824d129b8f25daf2115460 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:36.497684 1206 ts_tablet_manager.cc:1403] T 80098b7c24824d129b8f25daf2115460 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.004s
I20251214 20:52:36.444249 2323 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_0908bd3949a5414d86c41eedb660bfba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:36.498366 2323 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0908bd3949a5414d86c41eedb660bfba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.498451 1206 raft_consensus.cc:359] T 80098b7c24824d129b8f25daf2115460 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.498634 1206 raft_consensus.cc:385] T 80098b7c24824d129b8f25daf2115460 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.498703 1206 raft_consensus.cc:740] T 80098b7c24824d129b8f25daf2115460 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.498934 1206 consensus_queue.cc:260] T 80098b7c24824d129b8f25daf2115460 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.499441 1206 ts_tablet_manager.cc:1434] T 80098b7c24824d129b8f25daf2115460 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:36.444526 2322 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_d2030a17fe094e0b9a245170281ffaf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:36.499752 1206 tablet_bootstrap.cc:492] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:36.499893 2322 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2030a17fe094e0b9a245170281ffaf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.447036 2319 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_348add9458d84ff2ac739960e5f1407c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:36.502620 2319 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_348add9458d84ff2ac739960e5f1407c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:36.505125 848 consensus_peers.cc:597] T 8db4364b5023438bb43538855007c95e P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:36.516863 1206 tablet_bootstrap.cc:654] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.483297 3119 raft_consensus.cc:493] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.531347 3119 raft_consensus.cc:515] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.532199 3119 leader_election.cc:290] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.504241 2906 raft_consensus.cc:493] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.532800 2906 raft_consensus.cc:515] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.533313 2906 leader_election.cc:290] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.534965 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dbfe8ee68bf4a2c88489b1f81b36991" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.513125 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d745583012b4465090d85c75470e0bc2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.535641 1055 raft_consensus.cc:2468] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:36.536111 989 leader_election.cc:343] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.536768 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3de3864300a94e53a856c6a93e8d6e80" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:36.537278 989 leader_election.cc:343] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.538532 1203 ts_tablet_manager.cc:1403] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.166s user 0.002s sys 0.003s
I20251214 20:52:36.539633 1203 raft_consensus.cc:359] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.540084 1203 raft_consensus.cc:385] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.540190 1203 raft_consensus.cc:740] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.540493 1203 consensus_queue.cc:260] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.541118 1203 ts_tablet_manager.cc:1434] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:36.543840 1202 tablet_bootstrap.cc:492] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:36.544068 1202 ts_tablet_manager.cc:1403] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.132s user 0.008s sys 0.004s
I20251214 20:52:36.544901 1202 raft_consensus.cc:359] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.546986 1202 raft_consensus.cc:385] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.546164 2890 raft_consensus.cc:493] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.547762 2890 raft_consensus.cc:515] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.548403 2890 leader_election.cc:290] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.545189 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae948ffc81764692b92045f5befe0184" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:36.547591 1202 raft_consensus.cc:740] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.548707 903 raft_consensus.cc:3060] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.548866 1202 consensus_queue.cc:260] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.549161 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d01db63fd5d2474a9defdce0b98fc29d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.549324 915 raft_consensus.cc:2468] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 0.
I20251214 20:52:36.550766 1202 ts_tablet_manager.cc:1434] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:36.551165 1203 tablet_bootstrap.cc:492] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.551613 1202 tablet_bootstrap.cc:492] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:36.551666 712 leader_election.cc:343] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 25dcf491653147e7831217c1ec7a2e0c
I20251214 20:52:36.551828 712 leader_election.cc:304] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.552095 3272 raft_consensus.cc:2749] T 25dcf491653147e7831217c1ec7a2e0c P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.552578 3272 consensus_queue.cc:1048] T 088c72a6f5014b6fbc6ee060b6b7a590 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.552773 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f342dafae1d04be3ba38f7c0e4f5d058" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.552863 851 leader_election.cc:304] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.553200 3226 raft_consensus.cc:2804] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.553284 3226 raft_consensus.cc:493] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.553362 3226 raft_consensus.cc:3060] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.553321 712 leader_election.cc:304] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
W20251214 20:52:36.555585 848 consensus_peers.cc:597] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:36.555824 848 leader_election.cc:343] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.556470 1203 tablet_bootstrap.cc:654] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.558030 903 raft_consensus.cc:2468] T ae948ffc81764692b92045f5befe0184 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:36.558488 1206 tablet_bootstrap.cc:492] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:36.558650 1206 ts_tablet_manager.cc:1403] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20251214 20:52:36.559397 1206 raft_consensus.cc:359] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.559571 1206 raft_consensus.cc:385] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.559538 3226 raft_consensus.cc:515] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.559633 1206 raft_consensus.cc:740] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.559877 1206 consensus_queue.cc:260] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.560197 3226 leader_election.cc:290] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.561093 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d745583012b4465090d85c75470e0bc2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:36.561195 1206 ts_tablet_manager.cc:1434] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:36.562253 1206 tablet_bootstrap.cc:492] T a07d94d5a431488489ee9fbcc113a9d7 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:36.561942 848 leader_election.cc:343] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.564491 1206 tablet_bootstrap.cc:654] T a07d94d5a431488489ee9fbcc113a9d7 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.567013 1203 tablet_bootstrap.cc:492] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.567253 1203 ts_tablet_manager.cc:1403] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20251214 20:52:36.568449 1203 raft_consensus.cc:359] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.568671 1203 raft_consensus.cc:385] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.568768 1203 raft_consensus.cc:740] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.569048 1203 consensus_queue.cc:260] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.569324 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f342dafae1d04be3ba38f7c0e4f5d058" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.569664 1203 ts_tablet_manager.cc:1434] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:36.569525 1056 raft_consensus.cc:2468] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:36.570283 1203 tablet_bootstrap.cc:492] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.572439 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d745583012b4465090d85c75470e0bc2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:36.572593 1051 raft_consensus.cc:3060] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.573678 851 leader_election.cc:304] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.574566 2889 raft_consensus.cc:2804] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.574641 2889 raft_consensus.cc:493] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.574697 2889 raft_consensus.cc:3060] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.575434 1051 raft_consensus.cc:2468] T d745583012b4465090d85c75470e0bc2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.576259 1203 tablet_bootstrap.cc:654] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.577539 2889 raft_consensus.cc:515] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.578508 2889 leader_election.cc:290] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.579025 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f342dafae1d04be3ba38f7c0e4f5d058" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:36.580157 848 leader_election.cc:343] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.582099 2889 consensus_queue.cc:1048] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.582989 851 leader_election.cc:304] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.582983 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f342dafae1d04be3ba38f7c0e4f5d058" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:36.583174 1051 raft_consensus.cc:3060] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.583227 2889 consensus_queue.cc:1048] T d171301941fa43908e05984c3b016007 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.584412 3221 raft_consensus.cc:2804] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.584571 3221 raft_consensus.cc:697] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.584852 3221 consensus_queue.cc:237] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.586023 1051 raft_consensus.cc:2468] T f342dafae1d04be3ba38f7c0e4f5d058 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.590006 1202 tablet_bootstrap.cc:654] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.590893 851 leader_election.cc:304] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.592962 3108 raft_consensus.cc:2804] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.593199 3108 raft_consensus.cc:697] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.593506 3108 consensus_queue.cc:237] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.596534 1206 tablet_bootstrap.cc:492] T a07d94d5a431488489ee9fbcc113a9d7 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:36.596781 1206 ts_tablet_manager.cc:1403] T a07d94d5a431488489ee9fbcc113a9d7 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20251214 20:52:36.597592 1206 raft_consensus.cc:359] T a07d94d5a431488489ee9fbcc113a9d7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.597820 1206 raft_consensus.cc:385] T a07d94d5a431488489ee9fbcc113a9d7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.597894 1206 raft_consensus.cc:740] T a07d94d5a431488489ee9fbcc113a9d7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.598102 1206 consensus_queue.cc:260] T a07d94d5a431488489ee9fbcc113a9d7 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.599296 1206 ts_tablet_manager.cc:1434] T a07d94d5a431488489ee9fbcc113a9d7 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:36.599886 1206 tablet_bootstrap.cc:492] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:36.606294 848 consensus_peers.cc:597] T 28d34670fdce4242a3feccf830a12880 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:36.609421 1206 tablet_bootstrap.cc:654] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.611397 3283 raft_consensus.cc:493] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.611639 3283 raft_consensus.cc:515] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.612396 3283 leader_election.cc:290] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.613097 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67104e34df294ef5986171eca79cdc72" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:36.613631 989 leader_election.cc:343] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.614575 1203 tablet_bootstrap.cc:492] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.614755 1203 ts_tablet_manager.cc:1403] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.003s
I20251214 20:52:36.615226 3283 raft_consensus.cc:493] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.615374 3283 raft_consensus.cc:515] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.615942 3283 leader_election.cc:290] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.616458 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e68bf6b1a2974957adae70449334df64" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:36.616885 989 leader_election.cc:343] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.622587 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3de3864300a94e53a856c6a93e8d6e80" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.622838 1203 raft_consensus.cc:359] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.623026 1203 raft_consensus.cc:385] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.623092 1203 raft_consensus.cc:740] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.623310 1203 consensus_queue.cc:260] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.623656 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dbfe8ee68bf4a2c88489b1f81b36991" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.623971 1203 ts_tablet_manager.cc:1434] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:36.624276 1203 tablet_bootstrap.cc:492] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.625667 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67104e34df294ef5986171eca79cdc72" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.626370 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e68bf6b1a2974957adae70449334df64" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:36.627341 991 leader_election.cc:343] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.627452 991 leader_election.cc:304] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.627866 1203 tablet_bootstrap.cc:654] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:36.628014 991 leader_election.cc:343] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.628425 3283 raft_consensus.cc:2749] T 3de3864300a94e53a856c6a93e8d6e80 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.628404 991 leader_election.cc:304] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:36.628890 991 leader_election.cc:343] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.628976 991 leader_election.cc:304] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:36.629273 991 leader_election.cc:343] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.629350 991 leader_election.cc:304] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.629848 2906 raft_consensus.cc:2749] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.630187 2906 raft_consensus.cc:2749] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.630290 2906 raft_consensus.cc:2749] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.649902 1202 tablet_bootstrap.cc:492] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:36.650198 3111 consensus_queue.cc:1048] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.651854 1202 ts_tablet_manager.cc:1403] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.100s user 0.006s sys 0.006s
I20251214 20:52:36.652724 1202 raft_consensus.cc:359] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.652931 1202 raft_consensus.cc:385] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.652997 1202 raft_consensus.cc:740] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.653247 1202 consensus_queue.cc:260] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.653728 1202 ts_tablet_manager.cc:1434] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:36.655467 1206 tablet_bootstrap.cc:492] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:36.655534 3272 raft_consensus.cc:493] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.655673 1206 ts_tablet_manager.cc:1403] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.004s
I20251214 20:52:36.655758 3272 raft_consensus.cc:515] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.656481 3272 leader_election.cc:290] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:36.656422 1206 raft_consensus.cc:359] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.656612 1206 raft_consensus.cc:385] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.656661 1206 raft_consensus.cc:740] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.656843 1206 consensus_queue.cc:260] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.657395 1206 ts_tablet_manager.cc:1434] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:36.657788 1206 tablet_bootstrap.cc:492] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:36.659792 1206 tablet_bootstrap.cc:654] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.658282 3111 raft_consensus.cc:493] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.662123 3272 raft_consensus.cc:2804] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.662217 3272 raft_consensus.cc:493] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.662287 3272 raft_consensus.cc:3060] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.661906 1202 tablet_bootstrap.cc:492] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:36.662094 3111 raft_consensus.cc:515] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.663740 3111 leader_election.cc:290] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.664142 1202 tablet_bootstrap.cc:654] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.664151 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc8d725e6ac440899fc1cde2f7b546aa" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.664724 3272 raft_consensus.cc:515] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:36.664947 848 leader_election.cc:343] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.665401 3272 leader_election.cc:290] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:36.665668 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 112363us
I20251214 20:52:36.666462 712 leader_election.cc:304] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:36.666721 3272 raft_consensus.cc:2804] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.666962 3272 raft_consensus.cc:697] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:36.667245 3272 consensus_queue.cc:237] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.669824 2322 master_service.cc:954] ReplaceTablet: received request to replace tablet 0c62ffb71fff43d4812eb86f40ce1c58 from {username='slave'} at 127.0.0.1:41562
I20251214 20:52:36.672375 2321 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet c7248a2e8355425fa051a197160dbdbb
I20251214 20:52:36.674085 2321 catalog_manager.cc:6724] ReplaceTablet: tablet c7248a2e8355425fa051a197160dbdbb deleted and replaced by tablet e09e0481ebc64479ab0099093079ffe9
I20251214 20:52:36.674376 2321 master_service.cc:954] ReplaceTablet: received request to replace tablet b8727f302d35471eb2eac1e77b8d966b from {username='slave'} at 127.0.0.1:41562
I20251214 20:52:36.683107 775 raft_consensus.cc:3060] T 79e621f29c01448399d22615284fccd4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.684558 773 raft_consensus.cc:3060] T 8deabfdbd0674b2884ee5f82fa5e332f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.684863 2320 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 06ebe9f717bf43108748605cf4f961b7
I20251214 20:52:36.685392 2320 catalog_manager.cc:6724] ReplaceTablet: tablet 06ebe9f717bf43108748605cf4f961b7 deleted and replaced by tablet aadba4faca0c465f9eea004f93ee1190
I20251214 20:52:36.691828 1206 tablet_bootstrap.cc:492] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:36.692067 1206 ts_tablet_manager.cc:1403] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.004s
I20251214 20:52:36.693398 1206 raft_consensus.cc:359] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.693544 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc8d725e6ac440899fc1cde2f7b546aa" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.693293 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6eca34627bbc4a3f9915be72f3128370" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.693728 1051 raft_consensus.cc:2468] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:36.696019 1206 raft_consensus.cc:385] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251214 20:52:36.695468 713 leader_election.cc:343] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: d01db63fd5d2474a9defdce0b98fc29d
I20251214 20:52:36.700052 851 leader_election.cc:304] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.700441 3111 raft_consensus.cc:2804] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.700603 3111 raft_consensus.cc:493] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.700755 3111 raft_consensus.cc:3060] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.705387 3111 raft_consensus.cc:515] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.706598 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc8d725e6ac440899fc1cde2f7b546aa" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:36.707172 2906 raft_consensus.cc:493] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.707330 2906 raft_consensus.cc:515] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.707922 2906 leader_election.cc:290] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:36.708220 848 leader_election.cc:343] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.708768 3111 leader_election.cc:290] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.708935 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d535d8f71b44c81bbba6937e804d900" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:36.709370 989 leader_election.cc:343] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.710286 2316 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet fa141b2570514c6aacb2850e4bfd1869
I20251214 20:52:36.710826 2316 catalog_manager.cc:6724] ReplaceTablet: tablet fa141b2570514c6aacb2850e4bfd1869 deleted and replaced by tablet 02fc0fc09e8641f1bb05c312422e5971
I20251214 20:52:36.712711 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d535d8f71b44c81bbba6937e804d900" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.715798 1203 tablet_bootstrap.cc:492] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.715974 1203 ts_tablet_manager.cc:1403] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.092s user 0.016s sys 0.018s
I20251214 20:52:36.716661 1203 raft_consensus.cc:359] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:36.716768 991 leader_election.cc:343] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.716835 1203 raft_consensus.cc:385] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.716879 1203 raft_consensus.cc:740] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.716864 991 leader_election.cc:304] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.717044 1203 consensus_queue.cc:260] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.717298 2906 raft_consensus.cc:2749] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.717489 1203 ts_tablet_manager.cc:1434] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251214 20:52:36.717730 1203 tablet_bootstrap.cc:492] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.722194 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6eca34627bbc4a3f9915be72f3128370" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.723079 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d01db63fd5d2474a9defdce0b98fc29d" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999"
I20251214 20:52:36.723275 910 raft_consensus.cc:3060] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.726055 910 raft_consensus.cc:2468] T d01db63fd5d2474a9defdce0b98fc29d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074399864fb2470fa9e2ee87c1fe2212 in term 1.
I20251214 20:52:36.696468 1206 raft_consensus.cc:740] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.729883 712 leader_election.cc:304] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.730264 3272 raft_consensus.cc:2749] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.731597 1206 consensus_queue.cc:260] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:36.731755 2326 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69f9d5838e544002ab49f0ce21b6f7ef consensus state has no leader
I20251214 20:52:36.732151 712 leader_election.cc:304] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999; no voters: 63848a9fb78a4013a6f1657621272d75
I20251214 20:52:36.735278 773 raft_consensus.cc:3060] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.735491 2991 raft_consensus.cc:2804] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.735663 2991 raft_consensus.cc:697] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [term 1 LEADER]: Becoming Leader. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Running, Role: LEADER
I20251214 20:52:36.735817 3272 raft_consensus.cc:493] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.735917 2991 consensus_queue.cc:237] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.735931 3272 raft_consensus.cc:515] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:36.740000 3272 leader_election.cc:290] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:36.728318 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc8d725e6ac440899fc1cde2f7b546aa" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:36.739615 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e33ba1149a44c294885d76f676160a" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.742594 1056 raft_consensus.cc:3060] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.743144 1203 tablet_bootstrap.cc:654] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.748277 3272 raft_consensus.cc:493] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.748483 3272 raft_consensus.cc:515] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.749451 3272 leader_election.cc:290] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:36.749787 713 leader_election.cc:343] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 453cf734f8b34b59aa246d618ffeade5
I20251214 20:52:36.750336 753 tablet_service.cc:1552] Processing DeleteTablet for tablet c7248a2e8355425fa051a197160dbdbb with delete_type TABLET_DATA_DELETED (replaced by tablet e09e0481ebc64479ab0099093079ffe9) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:36.750447 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 06ebe9f717bf43108748605cf4f961b7 with delete_type TABLET_DATA_DELETED (replaced by tablet aadba4faca0c465f9eea004f93ee1190) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:36.750871 746 tablet_service.cc:1552] Processing DeleteTablet for tablet fa141b2570514c6aacb2850e4bfd1869 with delete_type TABLET_DATA_DELETED (replaced by tablet 02fc0fc09e8641f1bb05c312422e5971) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:36.751750 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet c7248a2e8355425fa051a197160dbdbb with delete_type TABLET_DATA_DELETED (replaced by tablet e09e0481ebc64479ab0099093079ffe9) from {username='slave'} at 127.0.0.1:49254
W20251214 20:52:36.752593 848 consensus_peers.cc:597] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:36.753600 1206 ts_tablet_manager.cc:1434] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.061s user 0.001s sys 0.001s
I20251214 20:52:36.754165 3226 consensus_queue.cc:1048] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.754338 2326 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_28893b766f8a4c93a8681c5e59752252"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:36.754932 2326 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28893b766f8a4c93a8681c5e59752252 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.755743 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e33ba1149a44c294885d76f676160a" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:36.755910 2320 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_31f83a8d780849d6b8b604d5223bd5a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:36.756140 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 06ebe9f717bf43108748605cf4f961b7 with delete_type TABLET_DATA_DELETED (replaced by tablet aadba4faca0c465f9eea004f93ee1190) from {username='slave'} at 127.0.0.1:49254
W20251214 20:52:36.756387 2320 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31f83a8d780849d6b8b604d5223bd5a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.756583 2328 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_581df19239f343f89fbe7e5c17d6d85c" } modify_external_catalogs: true
I20251214 20:52:36.756779 2328 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_581df19239f343f89fbe7e5c17d6d85c" } modify_external_catalogs: true
I20251214 20:52:36.757390 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet fa141b2570514c6aacb2850e4bfd1869 with delete_type TABLET_DATA_DELETED (replaced by tablet 02fc0fc09e8641f1bb05c312422e5971) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:36.758581 2319 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_ee7a207364d74e6b81d6a72e57cacbd5" } new_table_name: "default.Table_0d865be331094b9d8c407969618d12d4" modify_external_catalogs: true
I20251214 20:52:36.759999 2323 catalog_manager.cc:5654] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:36.760650 2323 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 4501fd5566a34e2a82009cc9ce6bdd37 (table default.Table_dce91ff3d9c74c7db1d400a24c4e550a [id=14f12b293a274ebeaf5bfefdff5b78cf]). Expected version 1 got 0
I20251214 20:52:36.761210 3305 raft_consensus.cc:493] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.761298 2316 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_7fbc660486544d9fb711dcdca0660237" } new_table_name: "default.Table_f9c22ae822b3401ab43ad74df0b0289c" modify_external_catalogs: true
I20251214 20:52:36.761348 3305 raft_consensus.cc:515] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.762377 3305 leader_election.cc:290] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:36.762859 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c07b455b12145e28ecff35f2732b840" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:36.763617 991 leader_election.cc:343] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.763693 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c07b455b12145e28ecff35f2732b840" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.763885 1206 tablet_bootstrap.cc:492] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:36.763936 2330 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_912c9fc9392c4d99aa8485af7917eb85" } new_table_name: "default.Table_6d745e9b39424aecbb85b0afa871ccd5" modify_external_catalogs: true
W20251214 20:52:36.764240 989 leader_election.cc:343] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.764328 989 leader_election.cc:304] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.764533 3305 raft_consensus.cc:2749] T 8c07b455b12145e28ecff35f2732b840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.769939 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 727e7b61ae3942098712720d344c5310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:36.773818 824 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:36.774933 2325 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_219b81378b7d4bccbd72bbb421faad87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:36.775489 2325 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_219b81378b7d4bccbd72bbb421faad87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.776827 712 leader_election.cc:304] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.777405 3039 raft_consensus.cc:2749] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:36.778163 712 leader_election.cc:343] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 453cf734f8b34b59aa246d618ffeade5
I20251214 20:52:36.778322 712 leader_election.cc:304] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.778590 3039 raft_consensus.cc:2749] T 453cf734f8b34b59aa246d618ffeade5 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.745682 1056 raft_consensus.cc:2468] T cc8d725e6ac440899fc1cde2f7b546aa P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.788703 851 leader_election.cc:304] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.792182 3287 raft_consensus.cc:2804] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.792413 3287 raft_consensus.cc:697] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.793354 3287 consensus_queue.cc:237] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:36.795449 2291 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 69f9d5838e544002ab49f0ce21b6f7ef (table default.Table_b77f9ca5f0f84a4db069e0a93db601e9 [id=4a1816b57ff84e459394e59b683d8ec6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 69f9d5838e544002ab49f0ce21b6f7ef consensus state has no leader
I20251214 20:52:36.797536 2322 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 0c62ffb71fff43d4812eb86f40ce1c58
I20251214 20:52:36.798089 2322 catalog_manager.cc:6724] ReplaceTablet: tablet 0c62ffb71fff43d4812eb86f40ce1c58 deleted and replaced by tablet cbf485f4814c4e6d88908d9541bc39a0
I20251214 20:52:36.798516 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c62ffb71fff43d4812eb86f40ce1c58 with delete_type TABLET_DATA_DELETED (replaced by tablet cbf485f4814c4e6d88908d9541bc39a0) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:36.798434 2322 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_66cfb79ea6504a84884abdbb156f707f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:36.799157 2322 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66cfb79ea6504a84884abdbb156f707f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.815380 2320 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_52200ef9aa154f64ae25af61268f6d6b" } modify_external_catalogs: true
I20251214 20:52:36.815642 2320 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_52200ef9aa154f64ae25af61268f6d6b" } modify_external_catalogs: true
I20251214 20:52:36.817901 2321 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet b8727f302d35471eb2eac1e77b8d966b
I20251214 20:52:36.818056 2323 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_c3bedc14bbf14c5988f9c60ceacf9d95" } modify_external_catalogs: true
I20251214 20:52:36.818190 2323 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_c3bedc14bbf14c5988f9c60ceacf9d95" } modify_external_catalogs: true
I20251214 20:52:36.818316 2321 catalog_manager.cc:6724] ReplaceTablet: tablet b8727f302d35471eb2eac1e77b8d966b deleted and replaced by tablet 9f629f102690489fb77e6d860067e0d5
I20251214 20:52:36.818640 2321 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_7bae2524af7c46b4a8c7df285f89970e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:36.819175 2321 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bae2524af7c46b4a8c7df285f89970e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:36.819397 2316 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7885f21a369461c8c80ff18d369e2c6 consensus state has no leader
I20251214 20:52:36.819700 2316 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_9060872cac6842548b9f1f80d16dbe19" } new_table_name: "default.Table_72013f5e5d4141f0aadb221a11e76eb7" modify_external_catalogs: true
W20251214 20:52:36.820700 2319 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 780be07e6f5d425d9605402b6db63012 consensus state has no leader
I20251214 20:52:36.820995 2319 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_403e5819717949e8bffa1ed844a68964" } modify_external_catalogs: true
I20251214 20:52:36.821130 2319 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_403e5819717949e8bffa1ed844a68964" } modify_external_catalogs: true
I20251214 20:52:36.822566 2328 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 99663ff0fe094849acefe03db8d43b50
W20251214 20:52:36.823223 2330 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c2a6c6c0b6e46bcb7658fbeced6f738 consensus state has no leader
I20251214 20:52:36.823570 2326 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_0a325ff1f4634ce7ada593df8f5a9fd6" } modify_external_catalogs: true
I20251214 20:52:36.823717 2326 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_0a325ff1f4634ce7ada593df8f5a9fd6" } modify_external_catalogs: true
I20251214 20:52:36.824812 962 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:36.835640 746 tablet_service.cc:1552] Processing DeleteTablet for tablet b8727f302d35471eb2eac1e77b8d966b with delete_type TABLET_DATA_DELETED (replaced by tablet 9f629f102690489fb77e6d860067e0d5) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:36.839490 773 raft_consensus.cc:3060] T 1d7aa508b11944aa8e268741c1998a9e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.841092 1103 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:36.841497 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 99663ff0fe094849acefe03db8d43b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:49254
W20251214 20:52:36.842087 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet c7248a2e8355425fa051a197160dbdbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7248a2e8355425fa051a197160dbdbb already in progress: creating tablet
W20251214 20:52:36.842422 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 06ebe9f717bf43108748605cf4f961b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06ebe9f717bf43108748605cf4f961b7 already in progress: creating tablet
W20251214 20:52:36.842648 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet fa141b2570514c6aacb2850e4bfd1869 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa141b2570514c6aacb2850e4bfd1869 already in progress: creating tablet
I20251214 20:52:36.843230 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c62ffb71fff43d4812eb86f40ce1c58 with delete_type TABLET_DATA_DELETED (replaced by tablet cbf485f4814c4e6d88908d9541bc39a0) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:36.841123 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet b8727f302d35471eb2eac1e77b8d966b with delete_type TABLET_DATA_DELETED (replaced by tablet 9f629f102690489fb77e6d860067e0d5) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:36.843768 3328 tablet_replica.cc:333] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:36.844040 3328 raft_consensus.cc:2243] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:36.844218 3328 raft_consensus.cc:2272] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:36.844331 3226 raft_consensus.cc:493] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.844498 3226 raft_consensus.cc:515] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.845139 3226 leader_election.cc:290] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.845160 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 8eb1006b0ba440dcba2f9de04dca2702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:36.845433 3328 ts_tablet_manager.cc:1916] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:36.846166 3329 tablet_replica.cc:333] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:36.846390 3329 raft_consensus.cc:2243] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:36.846560 3329 raft_consensus.cc:2272] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:36.848181 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet c7248a2e8355425fa051a197160dbdbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7248a2e8355425fa051a197160dbdbb already in progress: creating tablet
W20251214 20:52:36.848500 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 06ebe9f717bf43108748605cf4f961b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06ebe9f717bf43108748605cf4f961b7 already in progress: creating tablet
W20251214 20:52:36.848722 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet fa141b2570514c6aacb2850e4bfd1869 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa141b2570514c6aacb2850e4bfd1869 already in progress: creating tablet
I20251214 20:52:36.848937 3272 consensus_queue.cc:1048] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:36.849099 2991 raft_consensus.cc:493] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.849259 2991 raft_consensus.cc:515] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:36.849910 2991 leader_election.cc:290] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:36.850355 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9488c3505f5443998531cf749722be9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.850464 3329 ts_tablet_manager.cc:1916] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:36.850572 1056 raft_consensus.cc:2468] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:36.853735 2330 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_36e470f74bfc43368ea54b7bba3d0332"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:36.854369 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9488c3505f5443998531cf749722be9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:36.854436 2330 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36e470f74bfc43368ea54b7bba3d0332 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.854813 775 raft_consensus.cc:3060] T a07d94d5a431488489ee9fbcc113a9d7 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.854461 2322 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_f2b3131db41c42bf8c7ebb469e26ee87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:36.856175 2322 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2b3131db41c42bf8c7ebb469e26ee87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:36.857086 848 leader_election.cc:343] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.858546 851 leader_election.cc:304] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
W20251214 20:52:36.859368 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 99663ff0fe094849acefe03db8d43b50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99663ff0fe094849acefe03db8d43b50 already in progress: creating tablet
I20251214 20:52:36.859445 3221 raft_consensus.cc:2804] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.859525 3221 raft_consensus.cc:493] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.859584 3221 raft_consensus.cc:3060] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:36.859660 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 0c62ffb71fff43d4812eb86f40ce1c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c62ffb71fff43d4812eb86f40ce1c58 already in progress: creating tablet
W20251214 20:52:36.859863 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet b8727f302d35471eb2eac1e77b8d966b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8727f302d35471eb2eac1e77b8d966b already in progress: creating tablet
I20251214 20:52:36.860775 1202 tablet_bootstrap.cc:492] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:36.860909 1202 ts_tablet_manager.cc:1403] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.199s user 0.005s sys 0.001s
W20251214 20:52:36.860905 712 leader_election.cc:343] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 06381efe75284e649fd6d1a9903cfa9b
I20251214 20:52:36.861591 1202 raft_consensus.cc:359] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.861749 1202 raft_consensus.cc:385] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.861791 1202 raft_consensus.cc:740] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.861986 1202 consensus_queue.cc:260] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.862491 1202 ts_tablet_manager.cc:1434] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:36.862756 1202 tablet_bootstrap.cc:492] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:36.863546 1206 tablet_bootstrap.cc:654] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.864061 2320 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 52f5af48440b4d569939c163f365f6fe
I20251214 20:52:36.864642 3221 raft_consensus.cc:515] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.864893 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 52f5af48440b4d569939c163f365f6fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:33546
W20251214 20:52:36.866041 2316 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 719d157ac8f74eb7911df05ab5e4a3d5 consensus state has no leader
I20251214 20:52:36.866252 3221 leader_election.cc:290] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:36.866801 713 leader_election.cc:343] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 06381efe75284e649fd6d1a9903cfa9b
I20251214 20:52:36.866927 713 leader_election.cc:304] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:36.866946 2319 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 485ae1bdc1f344e08aadb4d00dc0a1f2
I20251214 20:52:36.867465 2991 raft_consensus.cc:2749] T 06381efe75284e649fd6d1a9903cfa9b P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:36.868978 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:36.869118 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9488c3505f5443998531cf749722be9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:36.869354 768 raft_consensus.cc:3060] T c8e33ba1149a44c294885d76f676160a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.869810 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:36.870064 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9488c3505f5443998531cf749722be9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:36.870265 1056 raft_consensus.cc:3060] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.872901 1056 raft_consensus.cc:2468] T c9488c3505f5443998531cf749722be9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.873445 851 leader_election.cc:304] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
W20251214 20:52:36.873934 848 consensus_peers.cc:597] T 825baba561ee4a8ebd44f75d8c51706a P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:36.875010 848 leader_election.cc:343] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.875613 3287 raft_consensus.cc:2804] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.875728 3287 raft_consensus.cc:697] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.875988 3287 consensus_queue.cc:237] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.877362 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 52f5af48440b4d569939c163f365f6fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:36.877461 3328 ts_tablet_manager.cc:1929] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:36.877568 3328 log.cc:1199] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/727e7b61ae3942098712720d344c5310
I20251214 20:52:36.877928 3328 ts_tablet_manager.cc:1950] T 727e7b61ae3942098712720d344c5310 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:36.880265 2288 catalog_manager.cc:4985] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): tablet 727e7b61ae3942098712720d344c5310 (table default.Table_828c0037b27e439a9dbb15886bb51344 [id=0a1ba44e3b1c4ea99e9badcb30bc72d0]) successfully deleted
W20251214 20:52:36.880708 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 0c62ffb71fff43d4812eb86f40ce1c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c62ffb71fff43d4812eb86f40ce1c58 already in progress: creating tablet
W20251214 20:52:36.881577 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet b8727f302d35471eb2eac1e77b8d966b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8727f302d35471eb2eac1e77b8d966b already in progress: creating tablet
W20251214 20:52:36.881839 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 52f5af48440b4d569939c163f365f6fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52f5af48440b4d569939c163f365f6fe already in progress: creating tablet
W20251214 20:52:36.882086 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 already in progress: creating tablet
I20251214 20:52:36.883283 894 tablet_service.cc:1552] Processing DeleteTablet for tablet c7248a2e8355425fa051a197160dbdbb with delete_type TABLET_DATA_DELETED (replaced by tablet e09e0481ebc64479ab0099093079ffe9) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:36.883718 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 06ebe9f717bf43108748605cf4f961b7 with delete_type TABLET_DATA_DELETED (replaced by tablet aadba4faca0c465f9eea004f93ee1190) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:36.884337 879 tablet_service.cc:1552] Processing DeleteTablet for tablet fa141b2570514c6aacb2850e4bfd1869 with delete_type TABLET_DATA_DELETED (replaced by tablet 02fc0fc09e8641f1bb05c312422e5971) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:36.884559 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c62ffb71fff43d4812eb86f40ce1c58 with delete_type TABLET_DATA_DELETED (replaced by tablet cbf485f4814c4e6d88908d9541bc39a0) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:36.884641 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 57a6232564194cbba12182f3a29c2ebb with delete_type TABLET_DATA_DELETED (replaced by tablet aaf060ab2f71429a85dda19669e69677) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:36.884531 2319 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_31aa6fe0cd4e481cb7a2d83d712d69f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:36.885054 885 tablet_service.cc:1552] Processing DeleteTablet for tablet b8727f302d35471eb2eac1e77b8d966b with delete_type TABLET_DATA_DELETED (replaced by tablet 9f629f102690489fb77e6d860067e0d5) from {username='slave'} at 127.0.0.1:56932
W20251214 20:52:36.885237 2319 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31aa6fe0cd4e481cb7a2d83d712d69f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.885545 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 99663ff0fe094849acefe03db8d43b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:36.887375 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 52f5af48440b4d569939c163f365f6fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:36.888039 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:36.889856 1203 tablet_bootstrap.cc:492] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.890018 1203 ts_tablet_manager.cc:1403] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.172s user 0.003s sys 0.000s
I20251214 20:52:36.890955 1203 raft_consensus.cc:359] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.891172 1203 raft_consensus.cc:385] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.891232 1203 raft_consensus.cc:740] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.891441 1203 consensus_queue.cc:260] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.891979 1203 ts_tablet_manager.cc:1434] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251214 20:52:36.892091 2326 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 5a43ee234b184275a6ab76b82fe5d283
I20251214 20:52:36.892304 1203 tablet_bootstrap.cc:492] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.892879 2323 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet cab1bed6016243bda0d9bd7f512884c8
W20251214 20:52:36.899858 848 consensus_peers.cc:597] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:36.904155 1203 tablet_bootstrap.cc:654] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.904569 755 tablet_service.cc:1552] Processing DeleteTablet for tablet cab1bed6016243bda0d9bd7f512884c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:36.906294 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a43ee234b184275a6ab76b82fe5d283 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:36.906662 885 tablet_service.cc:1552] Processing DeleteTablet for tablet cab1bed6016243bda0d9bd7f512884c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:36.907545 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a43ee234b184275a6ab76b82fe5d283 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:33546
W20251214 20:52:36.907720 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet cab1bed6016243bda0d9bd7f512884c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cab1bed6016243bda0d9bd7f512884c8 already in progress: creating tablet
W20251214 20:52:36.908174 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 5a43ee234b184275a6ab76b82fe5d283 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a43ee234b184275a6ab76b82fe5d283 already in progress: creating tablet
I20251214 20:52:36.917465 775 raft_consensus.cc:3060] T 6eca34627bbc4a3f9915be72f3128370 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.926286 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 99663ff0fe094849acefe03db8d43b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:33546
W20251214 20:52:36.928547 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 99663ff0fe094849acefe03db8d43b50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99663ff0fe094849acefe03db8d43b50 already in progress: creating tablet
I20251214 20:52:36.937220 2319 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_80443970562d4d4ca933febb7fe5708d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:36.938120 2319 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80443970562d4d4ca933febb7fe5708d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.944800 3226 raft_consensus.cc:493] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:36.944988 3226 raft_consensus.cc:515] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.945634 3226 leader_election.cc:290] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.946558 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34df3a34b15b4fe19bd12f785b07a033" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:36.946777 1053 raft_consensus.cc:2468] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:36.947945 1203 tablet_bootstrap.cc:492] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.948110 1203 ts_tablet_manager.cc:1403] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.003s
I20251214 20:52:36.948824 1203 raft_consensus.cc:359] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.948992 1203 raft_consensus.cc:385] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.949041 1203 raft_consensus.cc:740] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.949232 1203 consensus_queue.cc:260] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.950073 1203 ts_tablet_manager.cc:1434] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:36.950341 1203 tablet_bootstrap.cc:492] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.949558 851 leader_election.cc:304] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:36.950677 3226 raft_consensus.cc:2804] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:36.950754 3226 raft_consensus.cc:493] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:36.950812 3226 raft_consensus.cc:3060] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:36.952908 1203 tablet_bootstrap.cc:654] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.956043 3226 raft_consensus.cc:515] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.956730 3226 leader_election.cc:290] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:36.959311 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34df3a34b15b4fe19bd12f785b07a033" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:36.959949 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34df3a34b15b4fe19bd12f785b07a033" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:36.960376 1203 tablet_bootstrap.cc:492] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.960518 1203 ts_tablet_manager.cc:1403] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20251214 20:52:36.961195 1203 raft_consensus.cc:359] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.961356 1203 raft_consensus.cc:385] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.961407 1203 raft_consensus.cc:740] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.961795 1203 consensus_queue.cc:260] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.962661 2322 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_28893b766f8a4c93a8681c5e59752252" } new_table_name: "default.Table_d7da480ef0764d87b50b48556d4160e1" modify_external_catalogs: true
I20251214 20:52:36.962939 1203 ts_tablet_manager.cc:1434] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:36.963218 1203 tablet_bootstrap.cc:492] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.964807 1203 tablet_bootstrap.cc:654] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:36.965451 848 leader_election.cc:343] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:36.965873 848 leader_election.cc:343] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:36.967154 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "34df3a34b15b4fe19bd12f785b07a033" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:36.966967 2330 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_5dcef3898e0a47a5bbabc04825813686"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:36.967371 1053 raft_consensus.cc:3060] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:36.967862 2330 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dcef3898e0a47a5bbabc04825813686 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:36.970567 1053 raft_consensus.cc:2468] T 34df3a34b15b4fe19bd12f785b07a033 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:36.974700 851 leader_election.cc:304] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:36.977733 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet cab1bed6016243bda0d9bd7f512884c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:36.979031 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a43ee234b184275a6ab76b82fe5d283 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:36.979197 1203 tablet_bootstrap.cc:492] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:36.979326 1203 ts_tablet_manager.cc:1403] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20251214 20:52:36.980080 1203 raft_consensus.cc:359] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.980268 1203 raft_consensus.cc:385] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:36.980319 1203 raft_consensus.cc:740] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:36.980542 1203 consensus_queue.cc:260] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.981195 1203 ts_tablet_manager.cc:1434] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:36.984335 1203 tablet_bootstrap.cc:492] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:36.991524 3226 raft_consensus.cc:2804] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:36.992209 3226 raft_consensus.cc:697] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:36.992614 3226 consensus_queue.cc:237] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:36.993103 1203 tablet_bootstrap.cc:654] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:36.997875 776 raft_consensus.cc:3060] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.002205 770 raft_consensus.cc:3060] T 80098b7c24824d129b8f25daf2115460 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.004806 1203 tablet_bootstrap.cc:492] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.005102 1203 ts_tablet_manager.cc:1403] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.004s
I20251214 20:52:37.006531 1203 raft_consensus.cc:359] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.006863 1203 raft_consensus.cc:385] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.006985 1203 raft_consensus.cc:740] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.007516 1203 consensus_queue.cc:260] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.009624 3329 ts_tablet_manager.cc:1929] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:37.009747 1203 ts_tablet_manager.cc:1434] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251214 20:52:37.010028 3329 log.cc:1199] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/8eb1006b0ba440dcba2f9de04dca2702
I20251214 20:52:37.011107 3329 ts_tablet_manager.cc:1950] T 8eb1006b0ba440dcba2f9de04dca2702 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:37.027868 1202 tablet_bootstrap.cc:654] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.029503 2319 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_1774691d06214e3cbbb1d81b22cbf6c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:37.030356 2319 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1774691d06214e3cbbb1d81b22cbf6c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:37.044649 2326 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_ae25c010c04548198844cd6f5f3a5693" } new_table_name: "default.Table_45818a986b2f401e891f179268973f4c" modify_external_catalogs: true
I20251214 20:52:37.046020 1203 tablet_bootstrap.cc:492] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.048712 2316 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_d80c747f5a704a608b5e28b5e5d247b1" } new_table_name: "default.Table_dd3301152ada44209a885a8cffaf1fd9" modify_external_catalogs: true
I20251214 20:52:37.048761 2321 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_1d6a075fc8ca4e218a702db2b6715f5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:37.048990 2295 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 8eb1006b0ba440dcba2f9de04dca2702 (table default.Table_c2048db412744e2da280a49ec3d679b8 [id=e49aebe4f8214b9c879993f8cd2a1f6a]) successfully deleted
I20251214 20:52:37.049355 2320 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_d0c7cee7a77d47638ad0e5ea2f2c7315"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:37.049592 2321 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d6a075fc8ca4e218a702db2b6715f5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:37.049841 2320 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0c7cee7a77d47638ad0e5ea2f2c7315 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:37.050406 2328 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_80443970562d4d4ca933febb7fe5708d" } modify_external_catalogs: true
I20251214 20:52:37.050554 2325 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_92725fb3683841d7bd260746da84f2f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:37.050976 2325 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92725fb3683841d7bd260746da84f2f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:37.053956 3344 tablet.cc:1722] T 4501fd5566a34e2a82009cc9ce6bdd37 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251214 20:52:37.058790 2322 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 420c4887800948c088a591eadef5972d has no consensus state
I20251214 20:52:37.059104 2330 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_a54f22962823421ea5ab3033e0d9add0" } new_table_name: "default.Table_51d43737abff41769935a217bc776e71" modify_external_catalogs: true
I20251214 20:52:37.063498 3346 tablet.cc:1722] T 566c6e4761d5486d964e276d151e32b0 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:37.063637 2328 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_80443970562d4d4ca933febb7fe5708d" } modify_external_catalogs: true
I20251214 20:52:37.064772 3305 raft_consensus.cc:493] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.068159 3342 tablet_replica.cc:333] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:37.068385 3342 raft_consensus.cc:2243] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:37.068368 746 tablet_service.cc:1505] Processing CreateTablet for tablet 3adf510bc89a44009000d0e470c23fee (DEFAULT_TABLE table=default.Table_c15b44a5f25a4a38aa2d18ff7a59e5c6 [id=a4a47debc3894836b85ce71f4b4b0a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.068621 3342 pending_rounds.cc:70] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: Trying to abort 1 pending ops.
I20251214 20:52:37.068688 3342 pending_rounds.cc:77] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7232493798855913472 op_type: NO_OP noop_request { }
I20251214 20:52:37.068771 3342 raft_consensus.cc:2889] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251214 20:52:37.068845 3342 raft_consensus.cc:2272] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:37.069010 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3adf510bc89a44009000d0e470c23fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.072338 1202 tablet_bootstrap.cc:492] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.072315 755 tablet_service.cc:1505] Processing CreateTablet for tablet ccf2bd74820b4e9e843265343293cad1 (DEFAULT_TABLE table=default.Table_ac4df504ee284227bfd4ba2a23eaff65 [id=5f69f1d7797b4fff9030914e710a8fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.072453 1202 ts_tablet_manager.cc:1403] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.210s user 0.003s sys 0.000s
I20251214 20:52:37.072690 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf2bd74820b4e9e843265343293cad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.073303 745 tablet_service.cc:1505] Processing CreateTablet for tablet 8fcdb2ca52b14cf193d9ae87d0a50c56 (DEFAULT_TABLE table=default.Table_a1cc2f56735544ad8ae3425ad4d00fdb [id=bec715d329a74c7a8092f8e9c7f9c341]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.073662 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fcdb2ca52b14cf193d9ae87d0a50c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.075405 740 tablet_service.cc:1505] Processing CreateTablet for tablet 0207b2f46ca548c6881da6fd5dd17cf7 (DEFAULT_TABLE table=default.Table_d2030a17fe094e0b9a245170281ffaf8 [id=0a5914ec0b034205baeff19afb3ce648]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.075742 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0207b2f46ca548c6881da6fd5dd17cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.079393 752 tablet_service.cc:1505] Processing CreateTablet for tablet a34d3b8893224b5f94014c2da3d5ae65 (DEFAULT_TABLE table=default.Table_98b0ed640a93417ba49359781348b105 [id=5addd19777df4b968708a41c59a19651]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.079695 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34d3b8893224b5f94014c2da3d5ae65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.082449 3342 ts_tablet_manager.cc:1916] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:37.087589 1206 tablet_bootstrap.cc:492] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.087803 1206 ts_tablet_manager.cc:1403] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.330s user 0.004s sys 0.009s
I20251214 20:52:37.088824 1206 raft_consensus.cc:359] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.091061 1202 raft_consensus.cc:359] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.091346 1202 raft_consensus.cc:385] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.091440 1206 raft_consensus.cc:385] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.091506 1202 raft_consensus.cc:740] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.091589 1206 raft_consensus.cc:740] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.091789 1202 consensus_queue.cc:260] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.091924 1206 consensus_queue.cc:260] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.092487 1202 ts_tablet_manager.cc:1434] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251214 20:52:37.092689 1206 ts_tablet_manager.cc:1434] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251214 20:52:37.092836 1202 tablet_bootstrap.cc:492] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:37.094657 1202 tablet_bootstrap.cc:654] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.096809 1206 tablet_bootstrap.cc:492] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.098383 753 tablet_service.cc:1505] Processing CreateTablet for tablet aea9df19af594b4e81ccc024823baf4e (DEFAULT_TABLE table=default.Table_348add9458d84ff2ac739960e5f1407c [id=6273a95929ec41ffa33b77020a5c57a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.104892 1206 tablet_bootstrap.cc:654] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.105175 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea9df19af594b4e81ccc024823baf4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.109107 2889 consensus_queue.cc:1048] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.118041 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 3adf510bc89a44009000d0e470c23fee (DEFAULT_TABLE table=default.Table_c15b44a5f25a4a38aa2d18ff7a59e5c6 [id=a4a47debc3894836b85ce71f4b4b0a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.118518 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3adf510bc89a44009000d0e470c23fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.118870 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 0207b2f46ca548c6881da6fd5dd17cf7 (DEFAULT_TABLE table=default.Table_d2030a17fe094e0b9a245170281ffaf8 [id=0a5914ec0b034205baeff19afb3ce648]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.119216 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0207b2f46ca548c6881da6fd5dd17cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.123425 1023 tablet_service.cc:1505] Processing CreateTablet for tablet ccf2bd74820b4e9e843265343293cad1 (DEFAULT_TABLE table=default.Table_ac4df504ee284227bfd4ba2a23eaff65 [id=5f69f1d7797b4fff9030914e710a8fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.123803 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf2bd74820b4e9e843265343293cad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.125810 1028 tablet_service.cc:1505] Processing CreateTablet for tablet a34d3b8893224b5f94014c2da3d5ae65 (DEFAULT_TABLE table=default.Table_98b0ed640a93417ba49359781348b105 [id=5addd19777df4b968708a41c59a19651]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.126108 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34d3b8893224b5f94014c2da3d5ae65. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:37.126412 848 consensus_peers.cc:597] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.118362 1017 tablet_service.cc:1505] Processing CreateTablet for tablet aea9df19af594b4e81ccc024823baf4e (DEFAULT_TABLE table=default.Table_348add9458d84ff2ac739960e5f1407c [id=6273a95929ec41ffa33b77020a5c57a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.129953 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 8fcdb2ca52b14cf193d9ae87d0a50c56 (DEFAULT_TABLE table=default.Table_a1cc2f56735544ad8ae3425ad4d00fdb [id=bec715d329a74c7a8092f8e9c7f9c341]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.129981 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea9df19af594b4e81ccc024823baf4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.131419 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fcdb2ca52b14cf193d9ae87d0a50c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.133519 1202 tablet_bootstrap.cc:492] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.133699 1202 ts_tablet_manager.cc:1403] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20251214 20:52:37.134517 1202 raft_consensus.cc:359] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.134696 1202 raft_consensus.cc:385] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.134739 1202 raft_consensus.cc:740] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.134960 1202 consensus_queue.cc:260] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.137974 1027 tablet_service.cc:1505] Processing CreateTablet for tablet aadba4faca0c465f9eea004f93ee1190 (DEFAULT_TABLE table=default.Table_ae43058f9a39467b9919a9e1c335602f [id=7c91475975864cac93bfeb4d13173f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.138298 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadba4faca0c465f9eea004f93ee1190. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.142359 2906 raft_consensus.cc:493] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.142537 2906 raft_consensus.cc:515] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.143376 2906 leader_election.cc:290] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.064950 3305 raft_consensus.cc:515] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.144878 3305 leader_election.cc:290] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.145211 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a2fbec0543445b7856ef22c934c6840" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.145792 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ca12fc0de44082aa3d4fabc919c1dc" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.145884 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47ca12fc0de44082aa3d4fabc919c1dc" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.096406 3342 ts_tablet_manager.cc:1929] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:37.146175 3342 log.cc:1199] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/57a6232564194cbba12182f3a29c2ebb
I20251214 20:52:37.146291 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a2fbec0543445b7856ef22c934c6840" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:37.146361 991 leader_election.cc:343] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.146572 3342 ts_tablet_manager.cc:1950] T 57a6232564194cbba12182f3a29c2ebb P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
W20251214 20:52:37.146713 991 leader_election.cc:343] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:37.146883 989 leader_election.cc:343] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.147002 989 leader_election.cc:304] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.147372 3305 raft_consensus.cc:2749] T 47ca12fc0de44082aa3d4fabc919c1dc P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:37.147596 989 leader_election.cc:343] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.148324 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 472825da1a5243ab984f393ce98c8fe9 (DEFAULT_TABLE table=default.Table_eaeae6c30719483681416f2d216d1209 [id=68304e9495f44caab2e47d46335ab30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.148909 1019 tablet_service.cc:1505] Processing CreateTablet for tablet e09e0481ebc64479ab0099093079ffe9 (DEFAULT_TABLE table=default.Table_4bb4b496a62d4c89b7b4e783023dd5cd [id=144f1eb0434a44f49e52b76ba687525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.149434 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472825da1a5243ab984f393ce98c8fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.149927 3347 tablet.cc:1722] T 4501fd5566a34e2a82009cc9ce6bdd37 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:37.151419 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e09e0481ebc64479ab0099093079ffe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.147698 989 leader_election.cc:304] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.151777 3305 raft_consensus.cc:2749] T 0a2fbec0543445b7856ef22c934c6840 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.153542 1202 ts_tablet_manager.cc:1434] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.020s user 0.001s sys 0.002s
I20251214 20:52:37.155791 2889 raft_consensus.cc:493] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.155931 2889 raft_consensus.cc:515] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.156450 2889 leader_election.cc:290] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.157357 1206 tablet_bootstrap.cc:492] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.157370 1202 tablet_bootstrap.cc:492] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:37.157650 1206 ts_tablet_manager.cc:1403] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.002s
I20251214 20:52:37.158538 1206 raft_consensus.cc:359] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.158643 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd1161eebe884fe6b347bf517269c047" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.158816 1206 raft_consensus.cc:385] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.158937 1206 raft_consensus.cc:740] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.159330 1206 consensus_queue.cc:260] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.159526 894 tablet_service.cc:1505] Processing CreateTablet for tablet aea9df19af594b4e81ccc024823baf4e (DEFAULT_TABLE table=default.Table_348add9458d84ff2ac739960e5f1407c [id=6273a95929ec41ffa33b77020a5c57a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.160097 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aea9df19af594b4e81ccc024823baf4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.160288 885 tablet_service.cc:1505] Processing CreateTablet for tablet ccf2bd74820b4e9e843265343293cad1 (DEFAULT_TABLE table=default.Table_ac4df504ee284227bfd4ba2a23eaff65 [id=5f69f1d7797b4fff9030914e710a8fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.160398 1206 ts_tablet_manager.cc:1434] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251214 20:52:37.160583 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccf2bd74820b4e9e843265343293cad1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:37.160646 848 leader_election.cc:343] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.160921 1206 tablet_bootstrap.cc:492] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.159562 893 tablet_service.cc:1505] Processing CreateTablet for tablet 3adf510bc89a44009000d0e470c23fee (DEFAULT_TABLE table=default.Table_c15b44a5f25a4a38aa2d18ff7a59e5c6 [id=a4a47debc3894836b85ce71f4b4b0a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.163414 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3adf510bc89a44009000d0e470c23fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.164116 3305 raft_consensus.cc:493] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.164265 3305 raft_consensus.cc:515] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.171058 755 tablet_service.cc:1505] Processing CreateTablet for tablet 472825da1a5243ab984f393ce98c8fe9 (DEFAULT_TABLE table=default.Table_eaeae6c30719483681416f2d216d1209 [id=68304e9495f44caab2e47d46335ab30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.171569 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472825da1a5243ab984f393ce98c8fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.176251 1202 tablet_bootstrap.cc:654] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.176656 885 tablet_service.cc:1505] Processing CreateTablet for tablet 8fcdb2ca52b14cf193d9ae87d0a50c56 (DEFAULT_TABLE table=default.Table_a1cc2f56735544ad8ae3425ad4d00fdb [id=bec715d329a74c7a8092f8e9c7f9c341]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.177134 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fcdb2ca52b14cf193d9ae87d0a50c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.177446 893 tablet_service.cc:1505] Processing CreateTablet for tablet a34d3b8893224b5f94014c2da3d5ae65 (DEFAULT_TABLE table=default.Table_98b0ed640a93417ba49359781348b105 [id=5addd19777df4b968708a41c59a19651]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.177748 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34d3b8893224b5f94014c2da3d5ae65. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.163487 3349 tablet.cc:1722] T 566c6e4761d5486d964e276d151e32b0 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:37.186880 3305 leader_election.cc:290] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.184194 894 tablet_service.cc:1505] Processing CreateTablet for tablet 0207b2f46ca548c6881da6fd5dd17cf7 (DEFAULT_TABLE table=default.Table_d2030a17fe094e0b9a245170281ffaf8 [id=0a5914ec0b034205baeff19afb3ce648]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.188778 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99663ff0fe094849acefe03db8d43b50" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.189647 1202 tablet_bootstrap.cc:492] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.194736 2322 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_b72c21c5b00d430ea93eff8bf8bcb228" } modify_external_catalogs: true
I20251214 20:52:37.189658 879 tablet_service.cc:1505] Processing CreateTablet for tablet e09e0481ebc64479ab0099093079ffe9 (DEFAULT_TABLE table=default.Table_4bb4b496a62d4c89b7b4e783023dd5cd [id=144f1eb0434a44f49e52b76ba687525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.189832 878 tablet_service.cc:1505] Processing CreateTablet for tablet 472825da1a5243ab984f393ce98c8fe9 (DEFAULT_TABLE table=default.Table_eaeae6c30719483681416f2d216d1209 [id=68304e9495f44caab2e47d46335ab30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.189658 887 tablet_service.cc:1505] Processing CreateTablet for tablet aadba4faca0c465f9eea004f93ee1190 (DEFAULT_TABLE table=default.Table_ae43058f9a39467b9919a9e1c335602f [id=7c91475975864cac93bfeb4d13173f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.195976 1202 ts_tablet_manager.cc:1403] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20251214 20:52:37.196996 1202 raft_consensus.cc:359] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.197324 1202 raft_consensus.cc:385] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.197428 1202 raft_consensus.cc:740] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.197687 1202 consensus_queue.cc:260] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.198246 1202 ts_tablet_manager.cc:1434] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251214 20:52:37.194844 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet c7248a2e8355425fa051a197160dbdbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7248a2e8355425fa051a197160dbdbb already in progress: creating tablet
I20251214 20:52:37.199621 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99663ff0fe094849acefe03db8d43b50" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.202857 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e09e0481ebc64479ab0099093079ffe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.203816 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0207b2f46ca548c6881da6fd5dd17cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.203912 1202 tablet_bootstrap.cc:492] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:37.205783 1202 tablet_bootstrap.cc:654] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.207448 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472825da1a5243ab984f393ce98c8fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.211684 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadba4faca0c465f9eea004f93ee1190. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:37.216063 848 consensus_peers.cc:597] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.227265 3226 raft_consensus.cc:493] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.227449 3226 raft_consensus.cc:515] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.228155 3226 leader_election.cc:290] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.229010 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49778cfe80f6483389cfd421abee8ca7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:37.229687 848 leader_election.cc:343] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.247818 740 tablet_service.cc:1505] Processing CreateTablet for tablet e09e0481ebc64479ab0099093079ffe9 (DEFAULT_TABLE table=default.Table_4bb4b496a62d4c89b7b4e783023dd5cd [id=144f1eb0434a44f49e52b76ba687525c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.248313 3272 raft_consensus.cc:493] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.248425 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e09e0481ebc64479ab0099093079ffe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.248472 3272 raft_consensus.cc:515] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:37.249089 3272 leader_election.cc:290] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:37.189312 989 leader_election.cc:343] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:37.250875 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 61561.9us
I20251214 20:52:37.251809 1206 tablet_bootstrap.cc:654] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.252424 2906 raft_consensus.cc:493] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.252599 2906 raft_consensus.cc:515] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.252646 752 tablet_service.cc:1505] Processing CreateTablet for tablet 02fc0fc09e8641f1bb05c312422e5971 (DEFAULT_TABLE table=default.Table_47baf32513e6487fbd4ffde9bcba97ca [id=5ea1e541e5454f1ca96338573fda5705]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:37.253692 991 leader_election.cc:343] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:37.254779 712 leader_election.cc:343] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 3d949abef1cb41a9905482d81a98d2c1
I20251214 20:52:37.254923 991 leader_election.cc:304] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.255256 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02fc0fc09e8641f1bb05c312422e5971. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.259038 3039 raft_consensus.cc:493] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.259224 3039 raft_consensus.cc:515] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.259562 3305 raft_consensus.cc:2749] T 99663ff0fe094849acefe03db8d43b50 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.259830 3039 leader_election.cc:290] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:37.261395 712 leader_election.cc:343] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 58ac06f852fc4abea5cb213926002ca1
I20251214 20:52:37.264020 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd1161eebe884fe6b347bf517269c047" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.264240 1053 raft_consensus.cc:2468] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:37.264855 1050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49778cfe80f6483389cfd421abee8ca7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.264997 1050 raft_consensus.cc:2468] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:37.265878 848 consensus_peers.cc:597] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.267364 3305 raft_consensus.cc:493] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.267494 3305 raft_consensus.cc:515] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:37.267751 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 already in progress: creating tablet
I20251214 20:52:37.268214 745 tablet_service.cc:1505] Processing CreateTablet for tablet e3696c623bb0407a8fa4f7ee88409b04 (DEFAULT_TABLE table=default.Table_a5e4f40e603f4fd0a3d204f9c0298023 [id=36a5b0a44e8548e0a3c3f28975c4337d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.268743 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3696c623bb0407a8fa4f7ee88409b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.268882 1203 tablet_bootstrap.cc:654] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:37.270151 851 consensus_peers.cc:597] T f1eb4c3aa654480caaa69dbd5047365c P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1eb4c3aa654480caaa69dbd5047365c. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.270452 3226 consensus_queue.cc:1048] T 2f9def8e4fd449599820a63845397750 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.270617 851 leader_election.cc:304] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.270887 3226 consensus_queue.cc:1048] T 3bed2c326906486e99300e05033c9c0f P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.271159 3287 consensus_queue.cc:1048] T 1e185d107619414a8bcc15ad496547dc P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.271271 3226 raft_consensus.cc:2804] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:37.271353 3226 raft_consensus.cc:493] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:37.271458 3226 raft_consensus.cc:3060] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.273976 3226 raft_consensus.cc:515] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.274966 3226 leader_election.cc:290] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.275053 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd1161eebe884fe6b347bf517269c047" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:37.275580 851 leader_election.cc:304] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.276187 3111 raft_consensus.cc:2804] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:37.276288 3111 raft_consensus.cc:493] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:37.276396 3111 raft_consensus.cc:3060] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.277724 2906 leader_election.cc:290] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.278788 3111 raft_consensus.cc:515] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.278941 3305 leader_election.cc:290] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.279764 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62ac29a23c174ad9a3d93a10af7fecbe" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.280133 2322 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_b72c21c5b00d430ea93eff8bf8bcb228" } modify_external_catalogs: true
W20251214 20:52:37.280567 848 leader_election.cc:343] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.281109 3111 leader_election.cc:290] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:37.281646 991 leader_election.cc:343] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:37.282193 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 06ebe9f717bf43108748605cf4f961b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06ebe9f717bf43108748605cf4f961b7 already in progress: creating tablet
W20251214 20:52:37.282553 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet fa141b2570514c6aacb2850e4bfd1869 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa141b2570514c6aacb2850e4bfd1869 already in progress: creating tablet
I20251214 20:52:37.282859 753 tablet_service.cc:1505] Processing CreateTablet for tablet aadba4faca0c465f9eea004f93ee1190 (DEFAULT_TABLE table=default.Table_ae43058f9a39467b9919a9e1c335602f [id=7c91475975864cac93bfeb4d13173f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.283356 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aadba4faca0c465f9eea004f93ee1190. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:37.283877 713 consensus_peers.cc:597] T 0d7c7b3f62764ff497f4d70b53eed7de P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d7c7b3f62764ff497f4d70b53eed7de. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.284361 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0634838779854905b64dcdd028deb1e1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.284741 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "62ac29a23c174ad9a3d93a10af7fecbe" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.283207 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0634838779854905b64dcdd028deb1e1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.287545 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49778cfe80f6483389cfd421abee8ca7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:37.288156 848 leader_election.cc:343] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:37.288213 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 0c62ffb71fff43d4812eb86f40ce1c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c62ffb71fff43d4812eb86f40ce1c58 already in progress: creating tablet
I20251214 20:52:37.288774 2291 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 57a6232564194cbba12182f3a29c2ebb (table default.Table_9321256005b04155b2f5a451ce5065fe [id=c5a7a8665d6647bdb19ed6ef27907fee]) successfully deleted
W20251214 20:52:37.289799 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet b8727f302d35471eb2eac1e77b8d966b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8727f302d35471eb2eac1e77b8d966b already in progress: creating tablet
W20251214 20:52:37.290053 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 99663ff0fe094849acefe03db8d43b50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99663ff0fe094849acefe03db8d43b50 already in progress: creating tablet
W20251214 20:52:37.290632 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 52f5af48440b4d569939c163f365f6fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52f5af48440b4d569939c163f365f6fe already in progress: creating tablet
W20251214 20:52:37.290861 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 already in progress: creating tablet
W20251214 20:52:37.291225 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 5a43ee234b184275a6ab76b82fe5d283 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a43ee234b184275a6ab76b82fe5d283 already in progress: creating tablet
W20251214 20:52:37.291285 991 leader_election.cc:343] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.291373 1202 tablet_bootstrap.cc:492] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
W20251214 20:52:37.291416 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet cab1bed6016243bda0d9bd7f512884c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cab1bed6016243bda0d9bd7f512884c8 already in progress: creating tablet
I20251214 20:52:37.291532 1202 ts_tablet_manager.cc:1403] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.013s
W20251214 20:52:37.292898 2295 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 110898us
W20251214 20:52:37.293349 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 52f5af48440b4d569939c163f365f6fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52f5af48440b4d569939c163f365f6fe already in progress: creating tablet
I20251214 20:52:37.293537 1202 raft_consensus.cc:359] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:37.293833 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet cab1bed6016243bda0d9bd7f512884c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cab1bed6016243bda0d9bd7f512884c8 already in progress: creating tablet
W20251214 20:52:37.294029 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 5a43ee234b184275a6ab76b82fe5d283 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a43ee234b184275a6ab76b82fe5d283 already in progress: creating tablet
I20251214 20:52:37.294762 1202 raft_consensus.cc:385] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.294842 1202 raft_consensus.cc:740] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.295367 1202 consensus_queue.cc:260] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.297600 1202 ts_tablet_manager.cc:1434] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251214 20:52:37.299260 1202 tablet_bootstrap.cc:492] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:37.300841 989 leader_election.cc:343] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.301213 1202 tablet_bootstrap.cc:654] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.308281 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd1161eebe884fe6b347bf517269c047" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:37.308492 1047 raft_consensus.cc:3060] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.311682 887 tablet_service.cc:1505] Processing CreateTablet for tablet 02fc0fc09e8641f1bb05c312422e5971 (DEFAULT_TABLE table=default.Table_47baf32513e6487fbd4ffde9bcba97ca [id=5ea1e541e5454f1ca96338573fda5705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.312232 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02fc0fc09e8641f1bb05c312422e5971. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.322288 3039 raft_consensus.cc:493] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.324180 1047 raft_consensus.cc:2468] T bd1161eebe884fe6b347bf517269c047 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:37.324864 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49778cfe80f6483389cfd421abee8ca7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:37.325042 1049 raft_consensus.cc:3060] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.326442 879 tablet_service.cc:1505] Processing CreateTablet for tablet e3696c623bb0407a8fa4f7ee88409b04 (DEFAULT_TABLE table=default.Table_a5e4f40e603f4fd0a3d204f9c0298023 [id=36a5b0a44e8548e0a3c3f28975c4337d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.327085 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3696c623bb0407a8fa4f7ee88409b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.327822 989 leader_election.cc:304] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:37.328313 989 leader_election.cc:343] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.328437 989 leader_election.cc:304] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:37.330183 713 leader_election.cc:343] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 3d949abef1cb41a9905482d81a98d2c1
I20251214 20:52:37.330308 713 leader_election.cc:304] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
W20251214 20:52:37.330741 713 leader_election.cc:343] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 58ac06f852fc4abea5cb213926002ca1
I20251214 20:52:37.330830 713 leader_election.cc:304] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.331163 3272 raft_consensus.cc:2749] T 3d949abef1cb41a9905482d81a98d2c1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.331327 3272 raft_consensus.cc:2749] T 58ac06f852fc4abea5cb213926002ca1 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.336763 2328 catalog_manager.cc:5936] Not sending DeleteTablet requests; no consensus state for tablet 0234c79218b5493fb5845ab88091440e
I20251214 20:52:37.342628 851 leader_election.cc:304] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.353405 3226 raft_consensus.cc:2804] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:37.353686 3226 raft_consensus.cc:697] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:37.354003 3226 consensus_queue.cc:237] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.356491 2321 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_0945dd902049447298aef35db32f33ba" } modify_external_catalogs: true
I20251214 20:52:37.356772 2321 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_0945dd902049447298aef35db32f33ba" } modify_external_catalogs: true
I20251214 20:52:37.359614 2320 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_7f0a01f4a19a4b0380b81470a328ac40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:37.360327 2320 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f0a01f4a19a4b0380b81470a328ac40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:37.362047 3111 raft_consensus.cc:493] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.362267 3111 raft_consensus.cc:515] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.363149 3111 leader_election.cc:290] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.363193 3305 raft_consensus.cc:2749] T 62ac29a23c174ad9a3d93a10af7fecbe P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.363219 1049 raft_consensus.cc:2468] T 49778cfe80f6483389cfd421abee8ca7 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:37.363198 2906 raft_consensus.cc:2749] T 0634838779854905b64dcdd028deb1e1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.367167 1202 tablet_bootstrap.cc:492] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.367429 1202 ts_tablet_manager.cc:1403] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.003s
I20251214 20:52:37.368544 1202 raft_consensus.cc:359] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.368878 1202 raft_consensus.cc:385] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.369122 1202 raft_consensus.cc:740] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.369633 851 leader_election.cc:304] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.370149 3111 raft_consensus.cc:2804] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:37.370304 3111 raft_consensus.cc:697] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:37.370608 3111 consensus_queue.cc:237] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.380216 1202 consensus_queue.cc:260] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.381399 1202 ts_tablet_manager.cc:1434] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20251214 20:52:37.382277 1202 tablet_bootstrap.cc:492] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:37.383841 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532c44785f204f988518c6fd628f76ca" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.384074 1049 raft_consensus.cc:2468] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:37.385602 1202 tablet_bootstrap.cc:654] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:37.386166 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:36835, user_credentials={real_user=slave}} blocked reactor thread for 48903.3us
I20251214 20:52:37.389292 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 02fc0fc09e8641f1bb05c312422e5971 (DEFAULT_TABLE table=default.Table_47baf32513e6487fbd4ffde9bcba97ca [id=5ea1e541e5454f1ca96338573fda5705]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.389787 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02fc0fc09e8641f1bb05c312422e5971. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.392683 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532c44785f204f988518c6fd628f76ca" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:37.396250 848 leader_election.cc:343] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.396688 1017 tablet_service.cc:1505] Processing CreateTablet for tablet e3696c623bb0407a8fa4f7ee88409b04 (DEFAULT_TABLE table=default.Table_a5e4f40e603f4fd0a3d204f9c0298023 [id=36a5b0a44e8548e0a3c3f28975c4337d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.397125 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3696c623bb0407a8fa4f7ee88409b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.399263 3305 raft_consensus.cc:493] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.399407 3305 raft_consensus.cc:515] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.400008 3305 leader_election.cc:290] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.322484 3039 raft_consensus.cc:515] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } }
I20251214 20:52:37.400699 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c07966670d294a0495ec54d48c7f09cd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.400817 3039 leader_election.cc:290] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
W20251214 20:52:37.401335 991 leader_election.cc:343] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:37.404378 713 leader_election.cc:343] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: f84e9dfc23d34eb0a379e8ac5a442e59
I20251214 20:52:37.407987 3226 raft_consensus.cc:493] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.408182 3226 raft_consensus.cc:515] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.409034 3226 leader_election.cc:290] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.409571 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c2a6c6c0b6e46bcb7658fbeced6f738" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:37.410137 848 leader_election.cc:343] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.410724 851 leader_election.cc:304] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.411298 3221 raft_consensus.cc:2804] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:37.411442 3221 raft_consensus.cc:493] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:37.411536 3221 raft_consensus.cc:3060] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.414335 3221 raft_consensus.cc:515] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.414896 3221 leader_election.cc:290] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.416031 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532c44785f204f988518c6fd628f76ca" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:37.416628 848 leader_election.cc:343] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.419850 3305 raft_consensus.cc:493] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.420174 3305 raft_consensus.cc:515] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.420989 3305 leader_election.cc:290] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.421845 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c62ffb71fff43d4812eb86f40ce1c58" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:37.422394 991 leader_election.cc:343] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.423094 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c2a6c6c0b6e46bcb7658fbeced6f738" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.423326 1049 raft_consensus.cc:2468] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:37.424559 712 leader_election.cc:343] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: f84e9dfc23d34eb0a379e8ac5a442e59
I20251214 20:52:37.424743 712 leader_election.cc:304] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.425005 3039 raft_consensus.cc:2749] T f84e9dfc23d34eb0a379e8ac5a442e59 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.425259 3039 raft_consensus.cc:493] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.425457 3039 raft_consensus.cc:515] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.426167 3039 leader_election.cc:290] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.428122 1206 tablet_bootstrap.cc:492] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.429118 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3915abc53b084854912a8e4119eefc7e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.429418 1206 ts_tablet_manager.cc:1403] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.269s user 0.009s sys 0.019s
I20251214 20:52:37.430395 1206 raft_consensus.cc:359] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.430694 1206 raft_consensus.cc:385] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.431308 1206 raft_consensus.cc:740] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.431634 1206 consensus_queue.cc:260] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.435914 3039 raft_consensus.cc:493] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.436115 3039 raft_consensus.cc:515] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.437053 3039 leader_election.cc:290] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.436936 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "532c44785f204f988518c6fd628f76ca" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
W20251214 20:52:37.438323 713 leader_election.cc:343] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 0c973e0111804b8b869068bdcc5869bf
I20251214 20:52:37.437462 1051 raft_consensus.cc:3060] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.439642 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 38d688311da24b5cac50fe10ed9e6ab8 (DEFAULT_TABLE table=default.Table_81e31e2b08a1484c8adb4f139e0edfd3 [id=f6a2470a04cf40e0b3d04ec337fe328b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.440024 1206 ts_tablet_manager.cc:1434] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251214 20:52:37.440066 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38d688311da24b5cac50fe10ed9e6ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.440322 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 1986f856a8b44d62919976cd9c6b5f34 (DEFAULT_TABLE table=default.Table_26fdf6b490ab4f6cb488f406406a190c [id=d64c201020c34baf8cf82672f00aa29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.440610 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1986f856a8b44d62919976cd9c6b5f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.440675 1206 tablet_bootstrap.cc:492] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.442446 1206 tablet_bootstrap.cc:654] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.443246 3039 raft_consensus.cc:493] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.443387 3039 raft_consensus.cc:515] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.444051 3039 leader_election.cc:290] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.447448 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e4dc66681ef48479ae3cb95b091564a" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.452054 769 raft_consensus.cc:3060] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.454134 1206 tablet_bootstrap.cc:492] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.454417 1206 ts_tablet_manager.cc:1403] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20251214 20:52:37.454962 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 6b384bbc1f874d758dadd6932470713c (DEFAULT_TABLE table=default.Table_5b82f109385a42b0a04a5272bc5f0156 [id=40f4f5bacfea4828880fc526be3a06cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.455705 1206 raft_consensus.cc:359] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.456125 1206 raft_consensus.cc:385] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.456233 1206 raft_consensus.cc:740] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.456341 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b384bbc1f874d758dadd6932470713c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.456591 1206 consensus_queue.cc:260] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.457962 1206 ts_tablet_manager.cc:1434] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251214 20:52:37.458048 3221 raft_consensus.cc:493] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.458876 1206 tablet_bootstrap.cc:492] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.458272 3221 raft_consensus.cc:515] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.460314 3221 leader_election.cc:290] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.460723 1206 tablet_bootstrap.cc:654] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.461066 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43fe3bb1ed3941ce9078376f8f478408" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:37.463687 848 leader_election.cc:343] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.469520 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 32d5660cdab84d5e95925464efb5672b (DEFAULT_TABLE table=default.Table_0908bd3949a5414d86c41eedb660bfba [id=2917cad514c1478c8f7d47e3d2f3214a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.469908 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d5660cdab84d5e95925464efb5672b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.471803 1035 tablet_service.cc:1505] Processing CreateTablet for tablet bd77c84c26b44c85b9353b210212dbef (DEFAULT_TABLE table=default.Table_f9e379a41675483dbc3099b79604781a [id=3793f571e644488196a314e3e6adfcfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.472290 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd77c84c26b44c85b9353b210212dbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.478667 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c62ffb71fff43d4812eb86f40ce1c58" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.481132 1206 tablet_bootstrap.cc:492] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.481393 1206 ts_tablet_manager.cc:1403] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.001s
I20251214 20:52:37.482347 1206 raft_consensus.cc:359] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.482618 1206 raft_consensus.cc:385] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.482724 1206 raft_consensus.cc:740] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.483043 1206 consensus_queue.cc:260] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.483433 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c07966670d294a0495ec54d48c7f09cd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.483975 887 tablet_service.cc:1505] Processing CreateTablet for tablet 1986f856a8b44d62919976cd9c6b5f34 (DEFAULT_TABLE table=default.Table_26fdf6b490ab4f6cb488f406406a190c [id=d64c201020c34baf8cf82672f00aa29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.484027 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 6e8289d6605d4d35a4a475395a23d5e1 (DEFAULT_TABLE table=default.Table_c50d34acaf2e4ec8be9f4c60d90ec25c [id=2605ab6b89da49229adbc897734da196]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.484499 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1986f856a8b44d62919976cd9c6b5f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.484694 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e8289d6605d4d35a4a475395a23d5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.486989 1206 ts_tablet_manager.cc:1434] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:37.487602 1206 tablet_bootstrap.cc:492] T 76472ce9005b4bd2bbc51278816e8ae0 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.489744 1206 tablet_bootstrap.cc:654] T 76472ce9005b4bd2bbc51278816e8ae0 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.490165 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43fe3bb1ed3941ce9078376f8f478408" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.490852 1048 raft_consensus.cc:2468] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:37.489684 989 leader_election.cc:343] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.493141 1203 tablet_bootstrap.cc:492] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.493630 1203 ts_tablet_manager.cc:1403] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.448s user 0.003s sys 0.007s
I20251214 20:52:37.495147 1203 raft_consensus.cc:359] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.495520 1203 raft_consensus.cc:385] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.495618 1203 raft_consensus.cc:740] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.495834 1203 consensus_queue.cc:260] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.497140 1203 ts_tablet_manager.cc:1434] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251214 20:52:37.497541 2288 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42865, user_credentials={real_user=slave}} blocked reactor thread for 199858us
I20251214 20:52:37.497779 1203 tablet_bootstrap.cc:492] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.492457 989 leader_election.cc:304] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.499680 753 tablet_service.cc:1505] Processing CreateTablet for tablet 38d688311da24b5cac50fe10ed9e6ab8 (DEFAULT_TABLE table=default.Table_81e31e2b08a1484c8adb4f139e0edfd3 [id=f6a2470a04cf40e0b3d04ec337fe328b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.500079 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38d688311da24b5cac50fe10ed9e6ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.503423 746 tablet_service.cc:1505] Processing CreateTablet for tablet bd77c84c26b44c85b9353b210212dbef (DEFAULT_TABLE table=default.Table_f9e379a41675483dbc3099b79604781a [id=3793f571e644488196a314e3e6adfcfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.503821 754 tablet_service.cc:1505] Processing CreateTablet for tablet 827cddc5936346229e16611f0d73d587 (DEFAULT_TABLE table=default.Table_d050172b06964a17a220a95fb84d1081 [id=b2e03510ce354e1686aad6c081d52903]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.504277 755 tablet_service.cc:1505] Processing CreateTablet for tablet 6e8289d6605d4d35a4a475395a23d5e1 (DEFAULT_TABLE table=default.Table_c50d34acaf2e4ec8be9f4c60d90ec25c [id=2605ab6b89da49229adbc897734da196]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.505056 756 tablet_service.cc:1505] Processing CreateTablet for tablet 2021de48ee904cceb2b5e49131164689 (DEFAULT_TABLE table=default.Table_da9e4432a9254052ace01a09e425f97b [id=6203767083704e8b818e367287cfb130]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.505436 757 tablet_service.cc:1505] Processing CreateTablet for tablet 003bacf23a204e048509e0ceb6b80b9c (DEFAULT_TABLE table=default.Table_78f00e8dd81a485cae8269bac0d08866 [id=bc52dc01429e446b9726bc62fd9b24b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.505810 3305 raft_consensus.cc:2749] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.506163 740 tablet_service.cc:1505] Processing CreateTablet for tablet 1986f856a8b44d62919976cd9c6b5f34 (DEFAULT_TABLE table=default.Table_26fdf6b490ab4f6cb488f406406a190c [id=d64c201020c34baf8cf82672f00aa29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.506798 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd77c84c26b44c85b9353b210212dbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.507313 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827cddc5936346229e16611f0d73d587. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.507699 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 003bacf23a204e048509e0ceb6b80b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.508128 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2021de48ee904cceb2b5e49131164689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.511243 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e8289d6605d4d35a4a475395a23d5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.511667 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1986f856a8b44d62919976cd9c6b5f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.521317 1203 tablet_bootstrap.cc:654] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:37.522531 989 leader_election.cc:343] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.522639 989 leader_election.cc:304] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.522836 3305 raft_consensus.cc:2749] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.526057 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 2021de48ee904cceb2b5e49131164689 (DEFAULT_TABLE table=default.Table_da9e4432a9254052ace01a09e425f97b [id=6203767083704e8b818e367287cfb130]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.526515 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2021de48ee904cceb2b5e49131164689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.530652 2991 raft_consensus.cc:493] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.530874 2991 raft_consensus.cc:515] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.531011 1051 raft_consensus.cc:2468] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:37.531726 2991 leader_election.cc:290] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.537365 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 003bacf23a204e048509e0ceb6b80b9c (DEFAULT_TABLE table=default.Table_78f00e8dd81a485cae8269bac0d08866 [id=bc52dc01429e446b9726bc62fd9b24b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.538084 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 003bacf23a204e048509e0ceb6b80b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.539533 752 tablet_service.cc:1505] Processing CreateTablet for tablet 6b384bbc1f874d758dadd6932470713c (DEFAULT_TABLE table=default.Table_5b82f109385a42b0a04a5272bc5f0156 [id=40f4f5bacfea4828880fc526be3a06cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.539798 1035 tablet_service.cc:1505] Processing CreateTablet for tablet eb3ff5864734456090879ec7f06de288 (DEFAULT_TABLE table=default.Table_87794c3eaa7c4545acddac9ad4b4bc93 [id=59e0f0da4a1447a6957080f16c820e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.540083 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b384bbc1f874d758dadd6932470713c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.540257 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3ff5864734456090879ec7f06de288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.546131 1203 tablet_bootstrap.cc:492] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.546285 1203 ts_tablet_manager.cc:1403] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.001s
I20251214 20:52:37.547106 1203 raft_consensus.cc:359] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.549177 1203 raft_consensus.cc:385] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.550236 1203 raft_consensus.cc:740] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.550469 1203 consensus_queue.cc:260] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.551218 1203 ts_tablet_manager.cc:1434] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251214 20:52:37.551791 745 tablet_service.cc:1505] Processing CreateTablet for tablet 32d5660cdab84d5e95925464efb5672b (DEFAULT_TABLE table=default.Table_0908bd3949a5414d86c41eedb660bfba [id=2917cad514c1478c8f7d47e3d2f3214a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.552316 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d5660cdab84d5e95925464efb5672b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.555415 756 tablet_service.cc:1505] Processing CreateTablet for tablet eb3ff5864734456090879ec7f06de288 (DEFAULT_TABLE table=default.Table_87794c3eaa7c4545acddac9ad4b4bc93 [id=59e0f0da4a1447a6957080f16c820e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.555796 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3ff5864734456090879ec7f06de288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.562875 1203 tablet_bootstrap.cc:492] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.565764 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0ad1aa4193467db74cd707e7084b71" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:37.567592 713 consensus_peers.cc:597] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae948ffc81764692b92045f5befe0184. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.568254 1202 tablet_bootstrap.cc:492] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.568482 1202 ts_tablet_manager.cc:1403] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.186s user 0.003s sys 0.000s
I20251214 20:52:37.569229 1202 raft_consensus.cc:359] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.569456 1202 raft_consensus.cc:385] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.569538 1202 raft_consensus.cc:740] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.569767 1202 consensus_queue.cc:260] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.569916 1203 tablet_bootstrap.cc:654] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.570529 1202 ts_tablet_manager.cc:1434] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:37.570845 1202 tablet_bootstrap.cc:492] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:37.572520 1202 tablet_bootstrap.cc:654] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.591794 1203 tablet_bootstrap.cc:492] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.591951 1203 ts_tablet_manager.cc:1403] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.001s
I20251214 20:52:37.592602 1203 raft_consensus.cc:359] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.592780 1203 raft_consensus.cc:385] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.592828 1203 raft_consensus.cc:740] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.593039 1203 consensus_queue.cc:260] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.594012 1203 ts_tablet_manager.cc:1434] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:37.594619 2320 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_31f83a8d780849d6b8b604d5223bd5a2" } new_table_name: "default.Table_6d6fac7bc4f84ebc961567b02b145610" modify_external_catalogs: true
I20251214 20:52:37.594647 2319 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_b58a6dfd93fb438684f7564325241f3e" } modify_external_catalogs: true
I20251214 20:52:37.594919 2319 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_b58a6dfd93fb438684f7564325241f3e" } modify_external_catalogs: true
I20251214 20:52:37.595839 1203 tablet_bootstrap.cc:492] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.602882 1203 tablet_bootstrap.cc:654] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.607225 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 827cddc5936346229e16611f0d73d587 (DEFAULT_TABLE table=default.Table_d050172b06964a17a220a95fb84d1081 [id=b2e03510ce354e1686aad6c081d52903]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.573537 879 tablet_service.cc:1505] Processing CreateTablet for tablet 38d688311da24b5cac50fe10ed9e6ab8 (DEFAULT_TABLE table=default.Table_81e31e2b08a1484c8adb4f139e0edfd3 [id=f6a2470a04cf40e0b3d04ec337fe328b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.580412 893 tablet_service.cc:1505] Processing CreateTablet for tablet 32d5660cdab84d5e95925464efb5672b (DEFAULT_TABLE table=default.Table_0908bd3949a5414d86c41eedb660bfba [id=2917cad514c1478c8f7d47e3d2f3214a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.585880 885 tablet_service.cc:1505] Processing CreateTablet for tablet 6e8289d6605d4d35a4a475395a23d5e1 (DEFAULT_TABLE table=default.Table_c50d34acaf2e4ec8be9f4c60d90ec25c [id=2605ab6b89da49229adbc897734da196]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.585989 892 tablet_service.cc:1505] Processing CreateTablet for tablet 827cddc5936346229e16611f0d73d587 (DEFAULT_TABLE table=default.Table_d050172b06964a17a220a95fb84d1081 [id=b2e03510ce354e1686aad6c081d52903]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.586229 894 tablet_service.cc:1505] Processing CreateTablet for tablet bd77c84c26b44c85b9353b210212dbef (DEFAULT_TABLE table=default.Table_f9e379a41675483dbc3099b79604781a [id=3793f571e644488196a314e3e6adfcfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.586229 876 tablet_service.cc:1505] Processing CreateTablet for tablet 2021de48ee904cceb2b5e49131164689 (DEFAULT_TABLE table=default.Table_da9e4432a9254052ace01a09e425f97b [id=6203767083704e8b818e367287cfb130]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.580412 878 tablet_service.cc:1505] Processing CreateTablet for tablet 6b384bbc1f874d758dadd6932470713c (DEFAULT_TABLE table=default.Table_5b82f109385a42b0a04a5272bc5f0156 [id=40f4f5bacfea4828880fc526be3a06cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.586393 881 tablet_service.cc:1505] Processing CreateTablet for tablet 003bacf23a204e048509e0ceb6b80b9c (DEFAULT_TABLE table=default.Table_78f00e8dd81a485cae8269bac0d08866 [id=bc52dc01429e446b9726bc62fd9b24b3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251214 20:52:37.586819 848 consensus_peers.cc:597] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.608934 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38d688311da24b5cac50fe10ed9e6ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.587908 3226 raft_consensus.cc:493] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.609117 3226 raft_consensus.cc:515] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.609505 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e8289d6605d4d35a4a475395a23d5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.609818 3226 leader_election.cc:290] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.588373 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3915abc53b084854912a8e4119eefc7e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.610688 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d5660cdab84d5e95925464efb5672b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.610813 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827cddc5936346229e16611f0d73d587. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.615391 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827cddc5936346229e16611f0d73d587. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.619387 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd77c84c26b44c85b9353b210212dbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.623543 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2021de48ee904cceb2b5e49131164689. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.627378 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b384bbc1f874d758dadd6932470713c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.631419 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 003bacf23a204e048509e0ceb6b80b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.631603 1203 tablet_bootstrap.cc:492] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.631762 1203 ts_tablet_manager.cc:1403] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.000s
I20251214 20:52:37.632422 1203 raft_consensus.cc:359] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:37.632606 1203 raft_consensus.cc:385] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.632650 1203 raft_consensus.cc:740] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.632877 1203 consensus_queue.cc:260] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:37.633199 1206 tablet_bootstrap.cc:492] T 76472ce9005b4bd2bbc51278816e8ae0 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.633327 1206 ts_tablet_manager.cc:1403] T 76472ce9005b4bd2bbc51278816e8ae0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.146s user 0.043s sys 0.034s
I20251214 20:52:37.634032 1206 raft_consensus.cc:359] T 76472ce9005b4bd2bbc51278816e8ae0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.634183 1206 raft_consensus.cc:385] T 76472ce9005b4bd2bbc51278816e8ae0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.634227 1206 raft_consensus.cc:740] T 76472ce9005b4bd2bbc51278816e8ae0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.634438 1206 consensus_queue.cc:260] T 76472ce9005b4bd2bbc51278816e8ae0 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.634951 1206 ts_tablet_manager.cc:1434] T 76472ce9005b4bd2bbc51278816e8ae0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:37.635248 1206 tablet_bootstrap.cc:492] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.637257 1206 tablet_bootstrap.cc:654] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.637971 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147c985fa7c94f7a9af57917bf17b29f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.638159 1203 ts_tablet_manager.cc:1434] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20251214 20:52:37.638607 848 consensus_peers.cc:597] T 727e7b61ae3942098712720d344c5310 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 727e7b61ae3942098712720d344c5310. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.638459 1203 tablet_bootstrap.cc:492] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:37.640323 848 leader_election.cc:343] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.640508 1203 tablet_bootstrap.cc:654] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.588692 887 tablet_service.cc:1505] Processing CreateTablet for tablet eb3ff5864734456090879ec7f06de288 (DEFAULT_TABLE table=default.Table_87794c3eaa7c4545acddac9ad4b4bc93 [id=59e0f0da4a1447a6957080f16c820e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:37.591778 851 leader_election.cc:304] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.645071 3111 raft_consensus.cc:2804] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251214 20:52:37.645102 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 53355.8us
I20251214 20:52:37.645263 3111 raft_consensus.cc:493] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:37.645335 3111 raft_consensus.cc:3060] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.645929 851 leader_election.cc:304] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.646314 851 leader_election.cc:304] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.646467 3108 consensus_queue.cc:1048] T 1eff8a23d48c43e2bdbfb34f4bab6edf P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.592501 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "db0ad1aa4193467db74cd707e7084b71" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.648200 3111 raft_consensus.cc:515] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.648922 3111 leader_election.cc:290] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.649094 3111 raft_consensus.cc:2804] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:37.649150 3111 raft_consensus.cc:493] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:37.649204 3111 raft_consensus.cc:3060] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.651727 2777 raft_consensus.cc:2804] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:37.651913 2777 raft_consensus.cc:697] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:37.652174 2777 consensus_queue.cc:237] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.652643 3111 raft_consensus.cc:515] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.592964 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e4dc66681ef48479ae3cb95b091564a" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.654757 3111 leader_election.cc:290] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.658051 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3ff5864734456090879ec7f06de288. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:37.663100 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c2a6c6c0b6e46bcb7658fbeced6f738" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:37.663211 3039 raft_consensus.cc:493] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.663384 3039 raft_consensus.cc:515] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.664036 3039 leader_election.cc:290] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.664438 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43fe3bb1ed3941ce9078376f8f478408" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:37.667564 713 leader_election.cc:343] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 8eb1006b0ba440dcba2f9de04dca2702
W20251214 20:52:37.668123 848 leader_election.cc:343] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:37.668375 848 leader_election.cc:343] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:37.669014 712 leader_election.cc:343] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 0c973e0111804b8b869068bdcc5869bf
I20251214 20:52:37.669119 712 leader_election.cc:304] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.670329 3272 raft_consensus.cc:2749] T 0c973e0111804b8b869068bdcc5869bf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.670910 712 leader_election.cc:304] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.671353 2991 raft_consensus.cc:2749] T 3915abc53b084854912a8e4119eefc7e P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.671840 3111 raft_consensus.cc:493] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.672008 3111 raft_consensus.cc:515] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.672595 3111 leader_election.cc:290] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.674319 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eb1006b0ba440dcba2f9de04dca2702" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.674892 712 leader_election.cc:304] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.675288 3039 raft_consensus.cc:2749] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.676419 1206 tablet_bootstrap.cc:492] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.676565 1206 ts_tablet_manager.cc:1403] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.003s
I20251214 20:52:37.676779 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147c985fa7c94f7a9af57917bf17b29f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.676961 1047 raft_consensus.cc:2468] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:37.677335 1206 raft_consensus.cc:359] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.677491 1206 raft_consensus.cc:385] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.677541 1206 raft_consensus.cc:740] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.677732 1206 consensus_queue.cc:260] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:37.679102 848 consensus_peers.cc:597] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.679324 1206 ts_tablet_manager.cc:1434] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:37.680437 770 raft_consensus.cc:3060] T 76472ce9005b4bd2bbc51278816e8ae0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.680836 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "812642cfcff940f899ef9d3644909ec4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.693504 1206 tablet_bootstrap.cc:492] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.695340 1206 tablet_bootstrap.cc:654] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.698303 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43fe3bb1ed3941ce9078376f8f478408" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:37.698495 1048 raft_consensus.cc:3060] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.697304 712 leader_election.cc:304] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.699695 3359 raft_consensus.cc:2749] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.700352 712 leader_election.cc:304] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.700591 2991 raft_consensus.cc:2749] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.701292 1206 tablet_bootstrap.cc:492] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.701287 1048 raft_consensus.cc:2468] T 43fe3bb1ed3941ce9078376f8f478408 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:37.701421 1206 ts_tablet_manager.cc:1403] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20251214 20:52:37.701839 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "812642cfcff940f899ef9d3644909ec4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.702184 1048 raft_consensus.cc:2468] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:37.702224 848 leader_election.cc:343] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.702417 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c2a6c6c0b6e46bcb7658fbeced6f738" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:37.702639 1054 raft_consensus.cc:3060] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.703339 1206 raft_consensus.cc:359] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.705344 1054 raft_consensus.cc:2468] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
W20251214 20:52:37.706751 851 consensus_peers.cc:597] T 8eb1006b0ba440dcba2f9de04dca2702 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8eb1006b0ba440dcba2f9de04dca2702. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.707342 1206 raft_consensus.cc:385] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.707595 1206 raft_consensus.cc:740] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.707954 1206 consensus_queue.cc:260] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.711251 1206 ts_tablet_manager.cc:1434] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
W20251214 20:52:37.712675 848 consensus_peers.cc:597] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.712960 3226 consensus_queue.cc:1048] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.714203 1206 tablet_bootstrap.cc:492] T 2f9def8e4fd449599820a63845397750 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.714303 3287 consensus_queue.cc:1048] T d745583012b4465090d85c75470e0bc2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.714422 1203 tablet_bootstrap.cc:492] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.714581 1203 ts_tablet_manager.cc:1403] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.076s user 0.016s sys 0.010s
I20251214 20:52:37.714843 851 leader_election.cc:304] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.715049 3111 consensus_queue.cc:1048] T f342dafae1d04be3ba38f7c0e4f5d058 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.716297 1203 raft_consensus.cc:359] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.716486 1203 raft_consensus.cc:385] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.716543 1203 raft_consensus.cc:740] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.716774 2889 raft_consensus.cc:2804] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:37.716810 1203 consensus_queue.cc:260] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.716888 2889 raft_consensus.cc:493] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:37.716974 2889 raft_consensus.cc:3060] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.720896 2889 raft_consensus.cc:515] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.721586 2889 leader_election.cc:290] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.726418 1203 ts_tablet_manager.cc:1434] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251214 20:52:37.726791 1203 tablet_bootstrap.cc:492] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.730396 1203 tablet_bootstrap.cc:654] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.734706 851 leader_election.cc:304] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.735023 3111 raft_consensus.cc:2804] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:37.735177 3111 raft_consensus.cc:697] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:37.735460 3111 consensus_queue.cc:237] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.743523 1203 tablet_bootstrap.cc:492] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.743454 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147c985fa7c94f7a9af57917bf17b29f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:37.743703 1203 ts_tablet_manager.cc:1403] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
W20251214 20:52:37.744799 848 leader_election.cc:343] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.744757 1203 raft_consensus.cc:359] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.744938 1203 raft_consensus.cc:385] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.744999 1203 raft_consensus.cc:740] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
W20251214 20:52:37.745149 848 consensus_peers.cc:597] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.745429 1203 consensus_queue.cc:260] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.745703 851 leader_election.cc:304] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.745952 3111 raft_consensus.cc:2804] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:37.746026 3111 raft_consensus.cc:493] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:37.746088 3111 raft_consensus.cc:3060] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:37.746910 713 consensus_peers.cc:597] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 -> Peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Couldn't send request to peer 63848a9fb78a4013a6f1657621272d75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d01db63fd5d2474a9defdce0b98fc29d. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.747829 1203 ts_tablet_manager.cc:1434] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251214 20:52:37.748170 2323 catalog_manager.cc:5654] T a07d94d5a431488489ee9fbcc113a9d7 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } }
I20251214 20:52:37.749168 3111 raft_consensus.cc:515] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.750053 3111 leader_election.cc:290] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.750555 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "812642cfcff940f899ef9d3644909ec4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:37.750980 848 leader_election.cc:343] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.751579 851 leader_election.cc:304] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.752221 3226 raft_consensus.cc:2804] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:37.752389 3226 raft_consensus.cc:697] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:37.752710 3226 consensus_queue.cc:237] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.755388 1203 tablet_bootstrap.cc:492] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.756227 2323 catalog_manager.cc:5654] T 80098b7c24824d129b8f25daf2115460 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } }
I20251214 20:52:37.757302 1206 tablet_bootstrap.cc:654] T 2f9def8e4fd449599820a63845397750 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.759071 1203 tablet_bootstrap.cc:654] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:37.759119 2320 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d71efac30d2f4281a3df37e8306d9a9e has no consensus state
I20251214 20:52:37.764058 2320 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3" } modify_external_catalogs: true
I20251214 20:52:37.764462 2320 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3" } modify_external_catalogs: true
I20251214 20:52:37.765731 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 80098b7c24824d129b8f25daf2115460 (table default.Table_b14d897359c140bab6d04403fd59f31a [id=2a055be9028342be97f5c98f03f5a137]). Expected version 2 got 0
I20251214 20:52:37.766415 2323 catalog_manager.cc:5654] T e6bdfbe4d15345de8be5de00bfe4ca07 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } }
I20251214 20:52:37.767800 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_5de6830aaec645f2992e4cfaceb20c27 [id=9553b46856c94003a293135c814a7df7]). Expected version 2 got 0
I20251214 20:52:37.769081 1206 tablet_bootstrap.cc:492] T 2f9def8e4fd449599820a63845397750 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.771728 2323 catalog_manager.cc:5654] T 9f04336d2c84439b9003df19d3541273 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } }
I20251214 20:52:37.772660 1206 ts_tablet_manager.cc:1403] T 2f9def8e4fd449599820a63845397750 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20251214 20:52:37.779628 2323 catalog_manager.cc:5654] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } }
W20251214 20:52:37.780321 848 consensus_peers.cc:597] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.783215 1203 tablet_bootstrap.cc:492] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.783375 1203 ts_tablet_manager.cc:1403] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.003s
I20251214 20:52:37.784536 1203 raft_consensus.cc:359] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.784730 1203 raft_consensus.cc:385] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.784785 1203 raft_consensus.cc:740] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.784948 1203 consensus_queue.cc:260] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.785444 1203 ts_tablet_manager.cc:1434] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:37.785698 1203 tablet_bootstrap.cc:492] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.785811 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3a23561b9f4e48bbbd796e14fafec086 (table default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3 [id=f1338e81f4ec4dce87130f2ddbe98041]). Expected version 1 got 0
I20251214 20:52:37.780908 1206 raft_consensus.cc:359] T 2f9def8e4fd449599820a63845397750 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.787498 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "147c985fa7c94f7a9af57917bf17b29f" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:37.787703 1053 raft_consensus.cc:3060] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.787981 1203 tablet_bootstrap.cc:654] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.788538 1206 raft_consensus.cc:385] T 2f9def8e4fd449599820a63845397750 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.788640 1206 raft_consensus.cc:740] T 2f9def8e4fd449599820a63845397750 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.788872 1206 consensus_queue.cc:260] T 2f9def8e4fd449599820a63845397750 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.790584 1053 raft_consensus.cc:2468] T 147c985fa7c94f7a9af57917bf17b29f P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:37.789409 2323 catalog_manager.cc:5654] T f945814952b14131ba4e0c67a403a680 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:37.791404 3226 raft_consensus.cc:493] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.791605 3226 raft_consensus.cc:515] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.792568 3226 leader_election.cc:290] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.795226 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "812642cfcff940f899ef9d3644909ec4" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:37.795475 1053 raft_consensus.cc:3060] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.798135 1206 ts_tablet_manager.cc:1434] T 2f9def8e4fd449599820a63845397750 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251214 20:52:37.798476 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e5fff1fc84419ab3d3131cb3a12d92" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.798681 1053 raft_consensus.cc:2468] T 812642cfcff940f899ef9d3644909ec4 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:37.802614 1202 tablet_bootstrap.cc:492] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.802857 1202 ts_tablet_manager.cc:1403] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.232s user 0.015s sys 0.022s
I20251214 20:52:37.802948 1206 tablet_bootstrap.cc:492] T 825baba561ee4a8ebd44f75d8c51706a P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.791250 2323 catalog_manager.cc:5654] T 082ec70b8ee646b898d9201ad073d292 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:37.803714 1202 raft_consensus.cc:359] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:37.803966 1202 raft_consensus.cc:385] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.804068 1202 raft_consensus.cc:740] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.804050 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 082ec70b8ee646b898d9201ad073d292 (table default.Table_e3feb6d6cacb48dca8455eacf4cfb84b [id=8f720514e13b45689817a9811fb624d9]). Expected version 1 got 0
I20251214 20:52:37.804322 1202 consensus_queue.cc:260] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:37.804445 2323 catalog_manager.cc:5654] T 140667afe42f42659a8cd38ed2517201 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:37.805016 1202 ts_tablet_manager.cc:1434] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:37.805284 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 016581519f4c4e0d832b49346d55b41c (table default.Table_531eb871a88242c5b91423a2c6abc3b6 [id=9afdb569356648bb84089b1738dfc359]). Expected version 1 got 0
I20251214 20:52:37.805557 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 96a2b444608e42d687eeba2dc0ebb757 (table default.Table_90e94bab31e54a9cb8b567622aa4e37d [id=7c3d3542270e449ca954d068bece3491]). Expected version 1 got 0
I20251214 20:52:37.806104 1202 tablet_bootstrap.cc:492] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:37.807804 1202 tablet_bootstrap.cc:654] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.809571 2323 catalog_manager.cc:5654] T 6c1893b3761c49ceb7cd71b3bd9f2cb4 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:37.810361 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (table default.Table_fcafcf3c6ddb4457aea41320f7bf0b74 [id=4329d01e7f6147d0adad53fa86369b17]). Expected version 1 got 0
I20251214 20:52:37.810642 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet a5d28f09ad174936b47debace0599600 (table default.Table_93730665b1ab4686a0d3e54740fbddc8 [id=0b5581be22d345b3a0fd22c3dce2f207]). Expected version 1 got 0
I20251214 20:52:37.810930 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet f6d51d07f7e9450a844be9afddb60d03 (table default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]). Expected version 1 got 0
I20251214 20:52:37.811172 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet a4643effa71d4d18850913a59ae1c75d (table default.Table_679d53b1472341ef8612ca53ac1ebda2 [id=91f3b4dd176348a5aa38c6bdc417918f]). Expected version 3 got 0
I20251214 20:52:37.811609 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:37.812678 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 4501fd5566a34e2a82009cc9ce6bdd37 (table default.Table_dce91ff3d9c74c7db1d400a24c4e550a [id=14f12b293a274ebeaf5bfefdff5b78cf]). Expected version 1 got 0
I20251214 20:52:37.813098 2323 catalog_manager.cc:5654] T 28d34670fdce4242a3feccf830a12880 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:37.815469 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 69f9d5838e544002ab49f0ce21b6f7ef (table default.Table_b77f9ca5f0f84a4db069e0a93db601e9 [id=4a1816b57ff84e459394e59b683d8ec6]). Expected version 1 got 0
I20251214 20:52:37.816536 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738 (table default.Table_6d745e9b39424aecbb85b0afa871ccd5 [id=a56279b96c2d45af90277c7636192936]). Expected version 2 got 0
I20251214 20:52:37.818799 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 202b1d53cab644c98838b0ee81f29f22 (table default.Table_cddd59a230e04595bffdf9e0a0ed1471 [id=6094596f6c064c5bb1e20aa2b8767b9f]). Expected version 2 got 0
I20251214 20:52:37.819330 2906 raft_consensus.cc:493] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.819516 2906 raft_consensus.cc:515] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.819943 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 9bd89709abbc49809a2cdb6d0c29fdb8 (table default.Table_c3aa8e3c31cb41dc8281962e7b5c46e1 [id=687eb544b4114bad8d2deb76388e2022]). Expected version 1 got 0
I20251214 20:52:37.820243 2906 leader_election.cc:290] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.820685 2323 catalog_manager.cc:5654] T d424e3387698434f92e84eb1d43f5949 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:37.820912 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c5ee4114cb34540af2c5b332ebe82ab" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.820925 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c5ee4114cb34540af2c5b332ebe82ab" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:37.821579 991 leader_election.cc:343] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:37.821848 989 leader_election.cc:343] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.822031 989 leader_election.cc:304] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.822209 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d424e3387698434f92e84eb1d43f5949 (table default.Table_9098a11909c844dca6d4e80a284aa2e8 [id=1597e24e3fff4007ab66b44505c960d8]). Expected version 1 got 0
I20251214 20:52:37.822479 2906 raft_consensus.cc:2749] T 1c5ee4114cb34540af2c5b332ebe82ab P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.822485 2323 catalog_manager.cc:5654] T 4dcd511a5dca4a2cbda81fbc255ce354 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } }
I20251214 20:52:37.825414 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e5fff1fc84419ab3d3131cb3a12d92" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.825651 1056 raft_consensus.cc:2468] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:37.829638 3359 consensus_queue.cc:1048] T ae948ffc81764692b92045f5befe0184 P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.829923 3039 consensus_queue.cc:1048] T d01db63fd5d2474a9defdce0b98fc29d P 074399864fb2470fa9e2ee87c1fe2212 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:37.823706 2288 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d71efac30d2f4281a3df37e8306d9a9e (table default.Table_6d6fac7bc4f84ebc961567b02b145610 [id=49756930656347f9a28d186f60b347e3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d71efac30d2f4281a3df37e8306d9a9e has no consensus state
I20251214 20:52:37.830263 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]). Expected version 1 got 0
I20251214 20:52:37.833791 3108 consensus_queue.cc:1048] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.835399 851 leader_election.cc:304] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.835855 851 leader_election.cc:304] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.835983 1206 tablet_bootstrap.cc:654] T 825baba561ee4a8ebd44f75d8c51706a P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.836129 3108 raft_consensus.cc:2804] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:37.836221 3226 raft_consensus.cc:2804] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:37.836244 3108 raft_consensus.cc:697] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:37.836509 3226 raft_consensus.cc:697] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:37.836484 3108 consensus_queue.cc:237] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.837059 1202 tablet_bootstrap.cc:492] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.837033 3226 consensus_queue.cc:237] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.837221 1202 ts_tablet_manager.cc:1403] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20251214 20:52:37.837898 1202 raft_consensus.cc:359] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.838225 1202 raft_consensus.cc:385] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.838330 1202 raft_consensus.cc:740] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.838647 1202 consensus_queue.cc:260] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.839831 1202 ts_tablet_manager.cc:1434] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:37.840197 1202 tablet_bootstrap.cc:492] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:37.841729 1202 tablet_bootstrap.cc:654] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:37.842089 848 leader_election.cc:343] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.855571 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 67104e34df294ef5986171eca79cdc72 (table default.Table_27222912e9214b879d8a30975aba5688 [id=c0729ac770394d7b87ef36e53f85c151]). Expected version 3 got 0
I20251214 20:52:37.856607 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 526f28f9aaad445081d9d69cc20ed7b3 (table default.Table_b87980e2da6a4052b109938fcf0527d8 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 2 got 0
I20251214 20:52:37.857379 3272 raft_consensus.cc:493] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.857561 3272 raft_consensus.cc:515] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.858167 1203 tablet_bootstrap.cc:492] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.858304 1203 ts_tablet_manager.cc:1403] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.009s
I20251214 20:52:37.858299 3272 leader_election.cc:290] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.859037 1203 raft_consensus.cc:359] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.859246 1203 raft_consensus.cc:385] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.859299 1203 raft_consensus.cc:740] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.859476 1203 consensus_queue.cc:260] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.859961 1203 ts_tablet_manager.cc:1434] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:37.860241 1203 tablet_bootstrap.cc:492] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.861056 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e11723c042a4637839f1b13bed0935a" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:37.861891 1203 tablet_bootstrap.cc:654] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.862069 2323 catalog_manager.cc:5654] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } }
I20251214 20:52:37.863894 851 leader_election.cc:304] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:37.865083 3111 raft_consensus.cc:2804] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:37.865216 3111 raft_consensus.cc:493] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:37.865342 3111 raft_consensus.cc:3060] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.863070 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_72013f5e5d4141f0aadb221a11e76eb7 [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 3 got 0
I20251214 20:52:37.868341 3111 raft_consensus.cc:515] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.868710 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 0d0f7bd831b4474a90288ac7fa600436 (table default.Table_d8b86c6bc911454fac28b6384cf9ff3c [id=c97bcc9ac37c4413bae618ee1019e069]). Expected version 1 got 0
I20251214 20:52:37.869825 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e5fff1fc84419ab3d3131cb3a12d92" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:37.870442 848 leader_election.cc:343] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.871640 1206 tablet_bootstrap.cc:492] T 825baba561ee4a8ebd44f75d8c51706a P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.872243 3111 leader_election.cc:290] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.872923 3287 consensus_queue.cc:1048] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:37.877033 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet b87b0871c2454971a15dd5643c155694 (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:37.877451 2323 catalog_manager.cc:5654] T c6a8863ace4541aea7463af24db27142 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } }
I20251214 20:52:37.878255 2323 catalog_manager.cc:5654] T f11834cabdad445b8e53cd237d146282 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:37.878647 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet f11834cabdad445b8e53cd237d146282 (table default.Table_398f36b3acb440d88213e38f63740ca5 [id=954b3b4acdd94043b2608438d00483e7]). Expected version 1 got 0
I20251214 20:52:37.878957 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8a818ca230564b93ac165d43ff22deb1 (table default.Table_b289a25bf8174d9199c90212ce70271d [id=0a8ffa32e9ce4e3a8f9fa2afa4de69bb]). Expected version 1 got 0
I20251214 20:52:37.879262 2323 catalog_manager.cc:5654] T 161fa219cc2f4c44bbcfbdfe9dce9a1f P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } }
I20251214 20:52:37.879666 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:37.879915 2323 catalog_manager.cc:5654] T 4f1d0e555eee4c56be271d14c34335bd P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:37.880411 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet cc8d725e6ac440899fc1cde2f7b546aa (table default.Table_1052ac11eee9483aa13db090acff5b7a [id=978e62482f114f2b99e9707aaa9a5e32]). Expected version 1 got 0
I20251214 20:52:37.878223 770 raft_consensus.cc:3060] T 8eb1006b0ba440dcba2f9de04dca2702 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.883984 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "54e5fff1fc84419ab3d3131cb3a12d92" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:37.884253 1056 raft_consensus.cc:3060] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.888178 1056 raft_consensus.cc:2468] T 54e5fff1fc84419ab3d3131cb3a12d92 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:37.888278 1206 ts_tablet_manager.cc:1403] T 825baba561ee4a8ebd44f75d8c51706a P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.004s
I20251214 20:52:37.893078 1103 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
W20251214 20:52:37.893242 848 consensus_peers.cc:597] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:37.893337 1206 raft_consensus.cc:359] T 825baba561ee4a8ebd44f75d8c51706a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.893785 1206 raft_consensus.cc:385] T 825baba561ee4a8ebd44f75d8c51706a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.893958 1206 raft_consensus.cc:740] T 825baba561ee4a8ebd44f75d8c51706a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.894310 1206 consensus_queue.cc:260] T 825baba561ee4a8ebd44f75d8c51706a P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.896199 1206 ts_tablet_manager.cc:1434] T 825baba561ee4a8ebd44f75d8c51706a P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251214 20:52:37.899811 1202 tablet_bootstrap.cc:492] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.899986 1202 ts_tablet_manager.cc:1403] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.007s
I20251214 20:52:37.900691 1202 raft_consensus.cc:359] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.900878 1202 raft_consensus.cc:385] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.900934 1202 raft_consensus.cc:740] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.901134 1202 consensus_queue.cc:260] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.901360 1206 tablet_bootstrap.cc:492] T 054cfd587d19409fbdf7f4848bcaf3f9 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.902660 3272 raft_consensus.cc:493] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.902791 3272 raft_consensus.cc:515] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.903379 3272 leader_election.cc:290] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.906172 1202 ts_tablet_manager.cc:1434] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251214 20:52:37.906497 1202 tablet_bootstrap.cc:492] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
W20251214 20:52:37.910181 713 leader_election.cc:343] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 37a33cc9bb784f72956ac9f5493a0c3d
I20251214 20:52:37.910387 2906 raft_consensus.cc:493] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.910542 2906 raft_consensus.cc:515] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.911198 2906 leader_election.cc:290] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.910940 2323 catalog_manager.cc:5654] T 8deabfdbd0674b2884ee5f82fa5e332f P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63848a9fb78a4013a6f1657621272d75 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "63848a9fb78a4013a6f1657621272d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:37.911796 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a0f669313244e809c2868ca0ccfad65" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.912163 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 438b8ec5e9834442994966c8d83c1a31 (table default.Table_5cffb2d16a4847caa5cf6a2c4c238da8 [id=a8b3bbe26fc14b4bbc8a78c3d1f9df6d]). Expected version 1 got 0
W20251214 20:52:37.912338 991 leader_election.cc:343] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.912572 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet f8e98027ac3f4f36aded3aa5b47e6cad (table default.Table_5c7e3a65102c45a1a23d29b95427fae5 [id=db0d9e33bf054daabddadd1248243609]). Expected version 1 got 0
I20251214 20:52:37.913074 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet a12e8053c65343758d518ee840dbaee8 (table default.Table_cb922bf98617480aa9363c3f08d918f3 [id=e815640c7ece4361800a22bc749915e9]). Expected version 1 got 0
I20251214 20:52:37.913254 2906 raft_consensus.cc:493] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.913357 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a0f669313244e809c2868ca0ccfad65" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.913374 2906 raft_consensus.cc:515] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:37.913781 989 leader_election.cc:343] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.913894 2906 leader_election.cc:290] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:37.913870 989 leader_election.cc:304] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.914381 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47f610e6d08046a385c58274abab0e0b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:37.914757 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "47f610e6d08046a385c58274abab0e0b" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.915050 3371 raft_consensus.cc:2749] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.915226 3371 raft_consensus.cc:493] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.915313 3371 raft_consensus.cc:515] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.915803 3371 leader_election.cc:290] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:37.916500 991 leader_election.cc:343] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.917219 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f97e5bad46343098418e8e34334b892" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:37.917752 991 leader_election.cc:343] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.918404 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f97e5bad46343098418e8e34334b892" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:37.918776 989 leader_election.cc:343] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.918866 989 leader_election.cc:304] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.919291 3371 raft_consensus.cc:2749] T 47f610e6d08046a385c58274abab0e0b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:37.919797 989 leader_election.cc:343] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.919885 989 leader_election.cc:304] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:37.920259 3371 raft_consensus.cc:2749] T 2f97e5bad46343098418e8e34334b892 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:37.922582 2889 raft_consensus.cc:493] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.922719 2889 raft_consensus.cc:515] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.923295 2889 leader_election.cc:290] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.925272 768 raft_consensus.cc:3060] T 825baba561ee4a8ebd44f75d8c51706a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:37.927763 1206 tablet_bootstrap.cc:654] T 054cfd587d19409fbdf7f4848bcaf3f9 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.927837 1202 tablet_bootstrap.cc:654] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.930065 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4643effa71d4d18850913a59ae1c75d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:37.930143 1203 tablet_bootstrap.cc:492] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.930297 1203 ts_tablet_manager.cc:1403] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.004s
I20251214 20:52:37.931071 1203 raft_consensus.cc:359] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.931298 1203 raft_consensus.cc:385] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.931357 1203 raft_consensus.cc:740] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.931586 1203 consensus_queue.cc:260] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.932174 1203 ts_tablet_manager.cc:1434] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251214 20:52:37.932253 848 leader_election.cc:343] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:37.932447 1203 tablet_bootstrap.cc:492] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.932710 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e0d0e8f35bf44c9a8797b7507dcce3b9 (table default.Table_4649d8b3e43c42a492723fadd4217e4e [id=75cebf8d0aba4aa48c1be51b0abd6b38]). Expected version 2 got 0
I20251214 20:52:37.933080 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet e1dbaa39efa04086a22cc3908ed691d4 (table default.Table_8f897cb4591a4d5fadbf2b1b7ee7408d [id=4390cb960d564bb5bc7e0cefa5bf3004]). Expected version 1 got 0
I20251214 20:52:37.933751 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d39e245bc5a04c1186d0973645aa1b54 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 4 got 0
I20251214 20:52:37.934031 1203 tablet_bootstrap.cc:654] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.940476 1202 tablet_bootstrap.cc:492] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.940645 1202 ts_tablet_manager.cc:1403] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20251214 20:52:37.941341 1202 raft_consensus.cc:359] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.941504 1202 raft_consensus.cc:385] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.941557 1202 raft_consensus.cc:740] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.941761 1202 consensus_queue.cc:260] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.942310 1206 tablet_bootstrap.cc:492] T 054cfd587d19409fbdf7f4848bcaf3f9 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:37.942521 1206 ts_tablet_manager.cc:1403] T 054cfd587d19409fbdf7f4848bcaf3f9 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20251214 20:52:37.942616 1202 ts_tablet_manager.cc:1434] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:37.942903 1202 tablet_bootstrap.cc:492] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:37.944054 1203 tablet_bootstrap.cc:492] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.944200 1203 ts_tablet_manager.cc:1403] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20251214 20:52:37.944296 1202 tablet_bootstrap.cc:654] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.944912 1203 raft_consensus.cc:359] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.945070 1203 raft_consensus.cc:385] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.945125 1203 raft_consensus.cc:740] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.945338 1203 consensus_queue.cc:260] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.945787 1203 ts_tablet_manager.cc:1434] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:37.946079 1203 tablet_bootstrap.cc:492] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.947451 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet f891db00595b49ca9e6ae64721e4d744 (table default.Table_c404f3a3b71f4899a233f864e78a7d13 [id=203681807a4b40b0a1f69387cfecff38]). Expected version 1 got 0
I20251214 20:52:37.948179 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet bea3e9d7334345fa88ededf1be17a434 (table default.Table_27579b7155b140b3bdda5014c1eb57a8 [id=e24a6c9e5c9049edba5e42e22d22a831]). Expected version 1 got 0
I20251214 20:52:37.947153 1206 raft_consensus.cc:359] T 054cfd587d19409fbdf7f4848bcaf3f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.948627 1206 raft_consensus.cc:385] T 054cfd587d19409fbdf7f4848bcaf3f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.948832 1206 raft_consensus.cc:740] T 054cfd587d19409fbdf7f4848bcaf3f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.949172 1206 consensus_queue.cc:260] T 054cfd587d19409fbdf7f4848bcaf3f9 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.949906 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 96157aae81624a4ba9fcf8c3f142911f (table default.Table_adb418bf04534b868c044988fb97e906 [id=d7e403a944e94d0d8c9d6ba115723ea4]). Expected version 1 got 0
I20251214 20:52:37.959865 1203 tablet_bootstrap.cc:654] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.967041 2889 raft_consensus.cc:493] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:37.967273 2889 raft_consensus.cc:515] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.967826 2889 leader_election.cc:290] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:37.968060 1206 ts_tablet_manager.cc:1434] T 054cfd587d19409fbdf7f4848bcaf3f9 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251214 20:52:37.970198 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 780be07e6f5d425d9605402b6db63012 (table default.Table_0d865be331094b9d8c407969618d12d4 [id=b251a30885f6435fbfa430b6b50aee7a]). Expected version 1 got 0
I20251214 20:52:37.970355 1202 tablet_bootstrap.cc:492] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.970547 1202 ts_tablet_manager.cc:1403] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20251214 20:52:37.970920 1206 tablet_bootstrap.cc:492] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:37.971299 1202 raft_consensus.cc:359] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.971482 1202 raft_consensus.cc:385] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.971544 1202 raft_consensus.cc:740] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.971755 1202 consensus_queue.cc:260] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.972434 1202 ts_tablet_manager.cc:1434] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:37.972723 1202 tablet_bootstrap.cc:492] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:37.978077 1202 tablet_bootstrap.cc:654] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.972471 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet c7885f21a369461c8c80ff18d369e2c6 (table default.Table_f9c22ae822b3401ab43ad74df0b0289c [id=a20045f442604c45a67b9b06f1eb4610]). Expected version 2 got 0
I20251214 20:52:37.979898 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2398f176ee044c15841eaa40cb74974c (table default.Table_25630f29e4904aaf86f8f65078bcf81e [id=b1ee89b437fb42a4b5e2c3d386ee0284]). Expected version 1 got 0
I20251214 20:52:37.980031 1203 tablet_bootstrap.cc:492] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:37.980175 1203 ts_tablet_manager.cc:1403] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.007s
I20251214 20:52:37.980962 1203 raft_consensus.cc:359] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.981148 1203 raft_consensus.cc:385] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.981202 1203 raft_consensus.cc:740] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.981436 1203 consensus_queue.cc:260] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:37.982180 1206 tablet_bootstrap.cc:654] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.982411 1203 ts_tablet_manager.cc:1434] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:37.983191 1203 tablet_bootstrap.cc:492] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:37.983325 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 6302145a6e7b4297a58d9d7f3ad02dce (table default.Table_1be39fb576af4b2b9e2606b495619863 [id=54e0182cd3b9461aa593a11aa7709002]). Expected version 1 got 0
I20251214 20:52:37.983603 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 5ec7824f5eff4cc59c45acd3b3b3d194 (table default.Table_6cfa43d70d2f426fb58f8ef32476c3ac [id=be1be47d388347e3a6b7a80105688be3]). Expected version 1 got 0
I20251214 20:52:37.984964 1203 tablet_bootstrap.cc:654] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:37.986055 2323 catalog_manager.cc:5654] T db0ad1aa4193467db74cd707e7084b71 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:37.995770 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet db0ad1aa4193467db74cd707e7084b71 (table default.Table_5c7eb24742b84515af287c257637e51b [id=8acc06886693413682b246f25970fa5b]). Expected version 1 got 0
I20251214 20:52:37.996408 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 2807af0754914688bd7a090b1bc32eb1 (table default.Table_c6c0496d455a4db68c05c0ff8b06493d [id=9d1473df58ee486aa6a55893aca9d54c]). Expected version 3 got 0
I20251214 20:52:37.996722 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 4b5360e7ece94fd2a5a6512db6c6c12f (table default.Table_2ca54bf356e24a1ebca6f4a155b83044 [id=7f407f5abea945498cf939a23b0f9319]). Expected version 1 got 0
I20251214 20:52:37.996996 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 639cf3b7654a4ae896c6edd8b2ff1844 (table default.Table_15c1781afe3449ef87ef3c655ccca0df [id=02c00037bc944c00b0124d72af62e19b]). Expected version 1 got 0
I20251214 20:52:37.997540 1202 tablet_bootstrap.cc:492] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:37.997714 1202 ts_tablet_manager.cc:1403] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20251214 20:52:37.998287 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 3a0f669313244e809c2868ca0ccfad65 (table default.Table_d59e2b0d82384da68c1901ddaa419d00 [id=0c56f14b40ac442994e232aa348d64a2]). Expected version 1 got 0
I20251214 20:52:37.998565 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 894e3f27b7904db994711aa80a9b1600 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
I20251214 20:52:37.999341 1202 raft_consensus.cc:359] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:37.999538 1202 raft_consensus.cc:385] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:37.999598 1202 raft_consensus.cc:740] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:37.999820 1202 consensus_queue.cc:260] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.000274 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "336cb9d03fd44f70930662ee3c8d987a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.000432 1202 ts_tablet_manager.cc:1434] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:38.000736 1202 tablet_bootstrap.cc:492] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:38.001331 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 0dbfe8ee68bf4a2c88489b1f81b36991 (table default.Table_53bf327c948f4cebbe6096266ac301f8 [id=6a279e57008549c593df838211cc9cc1]). Expected version 1 got 0
I20251214 20:52:38.001629 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 6e005feeb95b48b9ad33612ab81d5bce (table default.Table_ec8c8f9185be4470a6580c4e141e2227 [id=8872bfac95c64cbba1ccb84d1d9f2483]). Expected version 2 got 0
I20251214 20:52:38.002565 1202 tablet_bootstrap.cc:654] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.003320 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet ba322cb6ee084bc39788ddd439f5f15d (table default.Table_196d0f488e0b42aabd518511a4289dc3 [id=2f57c28c9c624a6ca1986c8831b85169]). Expected version 1 got 0
I20251214 20:52:38.011521 2323 catalog_manager.cc:5654] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:38.016041 2906 raft_consensus.cc:493] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.016258 2906 raft_consensus.cc:515] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.016975 2906 leader_election.cc:290] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.017692 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad99467157df4c37bb9b518244a5dbd8" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.018102 1206 tablet_bootstrap.cc:492] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:38.018324 1206 ts_tablet_manager.cc:1403] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
W20251214 20:52:38.018352 991 leader_election.cc:343] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.018965 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad99467157df4c37bb9b518244a5dbd8" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.019042 3226 raft_consensus.cc:493] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.019210 3226 raft_consensus.cc:515] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:38.019677 989 leader_election.cc:343] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.019793 989 leader_election.cc:304] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.020193 2906 raft_consensus.cc:2749] T ad99467157df4c37bb9b518244a5dbd8 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.020638 1206 raft_consensus.cc:359] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.020889 1206 raft_consensus.cc:385] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.020991 1206 raft_consensus.cc:740] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.021224 1206 consensus_queue.cc:260] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.021503 1202 tablet_bootstrap.cc:492] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:38.021566 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8e11723c042a4637839f1b13bed0935a" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.021696 1202 ts_tablet_manager.cc:1403] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20251214 20:52:38.022030 1206 ts_tablet_manager.cc:1434] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251214 20:52:38.022392 1206 tablet_bootstrap.cc:492] T 4dcd511a5dca4a2cbda81fbc255ce354 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:38.022503 1202 raft_consensus.cc:359] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.022691 1202 raft_consensus.cc:385] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.022747 1202 raft_consensus.cc:740] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.022930 1202 consensus_queue.cc:260] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:38.024847 848 leader_election.cc:343] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.025372 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 566c6e4761d5486d964e276d151e32b0 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 2 got 0
I20251214 20:52:38.026187 1206 tablet_bootstrap.cc:654] T 4dcd511a5dca4a2cbda81fbc255ce354 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.026286 1202 ts_tablet_manager.cc:1434] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251214 20:52:38.026607 1202 tablet_bootstrap.cc:492] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:38.026821 2323 catalog_manager.cc:5654] T 8db4364b5023438bb43538855007c95e P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:38.028405 1202 tablet_bootstrap.cc:654] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.028681 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 8db4364b5023438bb43538855007c95e (table default.Table_80d7626996ca40979563d484ae097f08 [id=694ef81722d948638bbec8847c69d75a]). Expected version 1 got 0
I20251214 20:52:38.029948 2323 catalog_manager.cc:5654] T d2688515bd1441fbad7516f8343ca775 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:38.030493 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet d2688515bd1441fbad7516f8343ca775 (table default.Table_f1e9c435093240e4b1c1eb95635c8e66 [id=03997e2eab154ef18b2e3774b5984764]). Expected version 2 got 0
I20251214 20:52:38.032541 2323 catalog_manager.cc:5654] T 76472ce9005b4bd2bbc51278816e8ae0 P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:38.032975 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 76472ce9005b4bd2bbc51278816e8ae0 (table default.Table_9698b9206b934e4db2decdf9c0180838 [id=4b3f6c8d8cce4f7a96b2c677df6469ba]). Expected version 1 got 0
I20251214 20:52:38.033318 2323 catalog_manager.cc:5654] T 4e4dc66681ef48479ae3cb95b091564a P 63848a9fb78a4013a6f1657621272d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } }
I20251214 20:52:38.033737 2323 catalog_manager.cc:5746] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313) does not have the latest schema for tablet 59ccb6c3b35044dc913cce74b5802e38 (table default.Table_d61784eed0b84f8caa485dbe70cdc366 [id=b5622661ea9f40f7ac11f44ae92a5194]). Expected version 1 got 0
I20251214 20:52:38.033735 3226 leader_election.cc:290] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.035115 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f22ac24c38e4c64bb26c54ae4d87dfb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.036343 712 leader_election.cc:304] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.036904 3272 raft_consensus.cc:2749] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.038228 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4643effa71d4d18850913a59ae1c75d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.038419 1056 raft_consensus.cc:2468] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.039161 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "336cb9d03fd44f70930662ee3c8d987a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.039299 1056 raft_consensus.cc:2468] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.040362 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f22ac24c38e4c64bb26c54ae4d87dfb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.040508 1056 raft_consensus.cc:2468] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:38.044888 848 leader_election.cc:343] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.046166 712 leader_election.cc:343] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 37a33cc9bb784f72956ac9f5493a0c3d
I20251214 20:52:38.046295 712 leader_election.cc:304] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.046768 3272 raft_consensus.cc:2749] T 37a33cc9bb784f72956ac9f5493a0c3d P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.048096 3272 raft_consensus.cc:493] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.048228 3272 raft_consensus.cc:515] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.048835 3272 leader_election.cc:290] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.050194 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fd5566a34e2a82009cc9ce6bdd37" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.055917 851 leader_election.cc:304] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.056429 3111 raft_consensus.cc:2804] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.056541 3111 raft_consensus.cc:697] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.056805 3111 consensus_queue.cc:237] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.090875 851 leader_election.cc:304] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.091189 3221 raft_consensus.cc:2804] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.091264 3221 raft_consensus.cc:493] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.091318 3221 raft_consensus.cc:3060] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.092085 769 raft_consensus.cc:3060] T 2f9def8e4fd449599820a63845397750 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.093842 3221 raft_consensus.cc:515] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.094863 3221 leader_election.cc:290] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.095559 851 leader_election.cc:304] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.095860 3221 raft_consensus.cc:2804] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.095979 3221 raft_consensus.cc:493] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.096096 3221 raft_consensus.cc:3060] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.098749 3221 raft_consensus.cc:515] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.099603 3221 leader_election.cc:290] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.100672 851 leader_election.cc:304] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.102895 2889 raft_consensus.cc:2804] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.103047 2889 raft_consensus.cc:493] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.103194 2889 raft_consensus.cc:3060] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.104424 3039 raft_consensus.cc:493] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.104584 3039 raft_consensus.cc:515] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.104722 1206 tablet_bootstrap.cc:492] T 4dcd511a5dca4a2cbda81fbc255ce354 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:38.104839 1206 ts_tablet_manager.cc:1403] T 4dcd511a5dca4a2cbda81fbc255ce354 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.002s
I20251214 20:52:38.105226 3039 leader_election.cc:290] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.105535 1206 raft_consensus.cc:359] T 4dcd511a5dca4a2cbda81fbc255ce354 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.105788 2889 raft_consensus.cc:515] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.105710 1206 raft_consensus.cc:385] T 4dcd511a5dca4a2cbda81fbc255ce354 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.106029 1206 raft_consensus.cc:740] T 4dcd511a5dca4a2cbda81fbc255ce354 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.106215 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8db4364b5023438bb43538855007c95e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.106287 1206 consensus_queue.cc:260] T 4dcd511a5dca4a2cbda81fbc255ce354 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.106798 1206 ts_tablet_manager.cc:1434] T 4dcd511a5dca4a2cbda81fbc255ce354 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:38.107105 1206 tablet_bootstrap.cc:492] T f7f06eded6c340c58101fcc644ebcf50 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:38.107800 2906 raft_consensus.cc:493] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.107954 2906 raft_consensus.cc:515] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.108546 2906 leader_election.cc:290] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.108743 1206 tablet_bootstrap.cc:654] T f7f06eded6c340c58101fcc644ebcf50 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.109679 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b389a63a58d438db41a343db6e04a88" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.110041 911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4501fd5566a34e2a82009cc9ce6bdd37" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.110365 904 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8db4364b5023438bb43538855007c95e" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.107918 1202 tablet_bootstrap.cc:492] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:38.110582 1202 ts_tablet_manager.cc:1403] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.084s user 0.005s sys 0.002s
W20251214 20:52:38.110827 989 leader_election.cc:343] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.108263 3287 raft_consensus.cc:493] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.109261 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b389a63a58d438db41a343db6e04a88" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.111240 3287 raft_consensus.cc:515] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
W20251214 20:52:38.111593 991 leader_election.cc:343] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.111685 991 leader_election.cc:304] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.111937 3287 leader_election.cc:290] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:38.109526 3226 raft_consensus.cc:493] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.112120 3226 raft_consensus.cc:515] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.112660 2906 raft_consensus.cc:2749] T 6b389a63a58d438db41a343db6e04a88 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.112963 3226 leader_election.cc:290] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.113663 712 leader_election.cc:304] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.114208 3039 raft_consensus.cc:2749] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.114431 3111 raft_consensus.cc:493] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.114614 3111 raft_consensus.cc:515] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.115329 3111 leader_election.cc:290] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:38.115787 712 leader_election.cc:304] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.116182 3039 raft_consensus.cc:2749] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.116559 1202 raft_consensus.cc:359] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.116736 1202 raft_consensus.cc:385] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.116788 1202 raft_consensus.cc:740] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.117041 1202 consensus_queue.cc:260] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.117259 2889 leader_election.cc:290] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.117652 1202 ts_tablet_manager.cc:1434] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251214 20:52:38.118925 1202 tablet_bootstrap.cc:492] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:38.119554 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4643effa71d4d18850913a59ae1c75d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.119832 1056 raft_consensus.cc:3060] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.120784 1202 tablet_bootstrap.cc:654] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.123446 1056 raft_consensus.cc:2468] T a4643effa71d4d18850913a59ae1c75d P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.124191 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f22ac24c38e4c64bb26c54ae4d87dfb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.124261 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dbfe8ee68bf4a2c88489b1f81b36991" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.124389 1048 raft_consensus.cc:3060] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.124437 1053 raft_consensus.cc:2468] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.124706 2906 raft_consensus.cc:493] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.124831 2906 raft_consensus.cc:515] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.125501 2906 leader_election.cc:290] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.126121 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a41ed28272ad4e27bd2203dc4d3ad113" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:38.126653 991 leader_election.cc:343] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.127189 1048 raft_consensus.cc:2468] T 4f22ac24c38e4c64bb26c54ae4d87dfb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.127372 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a41ed28272ad4e27bd2203dc4d3ad113" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.128028 989 leader_election.cc:343] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.128131 989 leader_election.cc:304] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.128607 2906 raft_consensus.cc:2749] T a41ed28272ad4e27bd2203dc4d3ad113 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.128901 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "336cb9d03fd44f70930662ee3c8d987a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.129114 1056 raft_consensus.cc:3060] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.132645 1056 raft_consensus.cc:2468] T 336cb9d03fd44f70930662ee3c8d987a P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.136466 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a4643effa71d4d18850913a59ae1c75d" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.139798 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "336cb9d03fd44f70930662ee3c8d987a" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.140882 770 raft_consensus.cc:3060] T 4e4dc66681ef48479ae3cb95b091564a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.141435 851 leader_election.cc:304] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.142381 3111 raft_consensus.cc:2804] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.142503 3111 raft_consensus.cc:697] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.142716 3111 consensus_queue.cc:237] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.147399 851 leader_election.cc:304] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.147651 3111 raft_consensus.cc:2804] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.147774 3111 raft_consensus.cc:493] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.147857 3111 raft_consensus.cc:3060] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.150642 3111 raft_consensus.cc:515] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.151630 3111 leader_election.cc:290] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:38.152006 851 leader_election.cc:304] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.152581 3221 raft_consensus.cc:2804] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.152691 3221 raft_consensus.cc:697] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.152926 3221 consensus_queue.cc:237] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.153903 773 raft_consensus.cc:3060] T 8e11723c042a4637839f1b13bed0935a P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.153908 3108 consensus_queue.cc:1048] T c9488c3505f5443998531cf749722be9 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:38.154280 3108 consensus_queue.cc:1048] T 34df3a34b15b4fe19bd12f785b07a033 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:38.180744 1202 tablet_bootstrap.cc:492] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:38.181108 1202 ts_tablet_manager.cc:1403] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.000s
I20251214 20:52:38.181227 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dbfe8ee68bf4a2c88489b1f81b36991" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.181149 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f22ac24c38e4c64bb26c54ae4d87dfb" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.181735 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1def5f0de5ce4eb0bed36f626c4f0683" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.182128 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b482043af42485eb4c73d6eb864a963" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.182183 1202 raft_consensus.cc:359] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.182509 1202 raft_consensus.cc:385] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.182605 1202 raft_consensus.cc:740] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.182924 1202 consensus_queue.cc:260] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.133430 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1def5f0de5ce4eb0bed36f626c4f0683" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.183337 1048 raft_consensus.cc:2468] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.183519 1202 ts_tablet_manager.cc:1434] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:38.133507 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b482043af42485eb4c73d6eb864a963" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.183804 1202 tablet_bootstrap.cc:492] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:38.183825 1053 raft_consensus.cc:2468] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.147224 2906 raft_consensus.cc:493] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.171262 3371 raft_consensus.cc:493] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.186503 3371 raft_consensus.cc:515] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.179298 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dbfe8ee68bf4a2c88489b1f81b36991" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.187151 1202 tablet_bootstrap.cc:654] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.187220 1049 raft_consensus.cc:3060] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.190248 1049 raft_consensus.cc:2468] T 0dbfe8ee68bf4a2c88489b1f81b36991 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.190881 3371 leader_election.cc:290] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.191642 3111 raft_consensus.cc:493] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.191800 3111 raft_consensus.cc:515] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.192402 3111 leader_election.cc:290] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.196053 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5e0b89d02c04625b8e31910d60d5d40" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.197631 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5e0b89d02c04625b8e31910d60d5d40" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.197755 1202 tablet_bootstrap.cc:492] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:38.198050 1202 ts_tablet_manager.cc:1403] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
W20251214 20:52:38.198074 991 leader_election.cc:343] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.198336 2906 raft_consensus.cc:515] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.198999 1202 raft_consensus.cc:359] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.199256 1202 raft_consensus.cc:385] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.199339 1202 raft_consensus.cc:740] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.199478 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "016581519f4c4e0d832b49346d55b41c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.199590 1202 consensus_queue.cc:260] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:38.199925 989 leader_election.cc:343] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.200034 989 leader_election.cc:304] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.200557 3305 raft_consensus.cc:2749] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.200943 1202 ts_tablet_manager.cc:1434] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20251214 20:52:38.200950 989 leader_election.cc:343] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.201370 2906 leader_election.cc:290] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.201707 1202 tablet_bootstrap.cc:492] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:38.202068 903 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "016581519f4c4e0d832b49346d55b41c" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:38.202704 991 leader_election.cc:343] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.202834 991 leader_election.cc:304] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.203711 2906 raft_consensus.cc:2749] T 016581519f4c4e0d832b49346d55b41c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.208088 2906 raft_consensus.cc:493] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.208318 2906 raft_consensus.cc:515] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.209465 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9c383e343544c83ada89bea6ab20e94" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.210063 989 leader_election.cc:343] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.210124 2906 leader_election.cc:290] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.210516 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9c383e343544c83ada89bea6ab20e94" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:38.211078 991 leader_election.cc:343] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.211218 991 leader_election.cc:304] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.211635 2906 raft_consensus.cc:2749] T a9c383e343544c83ada89bea6ab20e94 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.211974 1202 tablet_bootstrap.cc:654] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.226979 962 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:38.227603 1202 tablet_bootstrap.cc:492] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:38.227823 1202 ts_tablet_manager.cc:1403] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.002s
I20251214 20:52:38.228690 1202 raft_consensus.cc:359] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.228956 1202 raft_consensus.cc:385] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.229046 1202 raft_consensus.cc:740] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.229312 1202 consensus_queue.cc:260] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.229966 1202 ts_tablet_manager.cc:1434] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:38.230275 1202 tablet_bootstrap.cc:492] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:38.231984 1202 tablet_bootstrap.cc:654] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.235816 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25180d5352204cdca3c49f907a587cb2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.236095 1053 raft_consensus.cc:2468] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.238219 851 leader_election.cc:304] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.238588 3287 raft_consensus.cc:2804] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.238706 3287 raft_consensus.cc:493] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.238766 851 leader_election.cc:304] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.238767 3287 raft_consensus.cc:3060] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.239161 851 leader_election.cc:304] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.239501 851 leader_election.cc:304] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.240494 851 leader_election.cc:304] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.241741 3287 raft_consensus.cc:515] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.242779 2889 raft_consensus.cc:2804] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.242861 2889 raft_consensus.cc:493] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.242928 2889 raft_consensus.cc:3060] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.243208 3111 raft_consensus.cc:2804] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.243507 3108 raft_consensus.cc:2804] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.244143 3221 raft_consensus.cc:2804] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.244241 3221 raft_consensus.cc:493] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.244302 3221 raft_consensus.cc:3060] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.247932 2889 raft_consensus.cc:515] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.248514 2889 leader_election.cc:290] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:38.251627 3287 leader_election.cc:290] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.253705 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dbfe8ee68bf4a2c88489b1f81b36991" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.254237 1203 tablet_bootstrap.cc:492] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.254392 1203 ts_tablet_manager.cc:1403] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.271s user 0.028s sys 0.059s
I20251214 20:52:38.255182 774 raft_consensus.cc:3060] T db0ad1aa4193467db74cd707e7084b71 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.255252 1203 raft_consensus.cc:359] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.255422 1203 raft_consensus.cc:385] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.255466 1203 raft_consensus.cc:740] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.255645 1203 consensus_queue.cc:260] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.256135 1203 ts_tablet_manager.cc:1434] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:38.256979 1206 tablet_bootstrap.cc:492] T f7f06eded6c340c58101fcc644ebcf50 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:38.257221 1206 ts_tablet_manager.cc:1403] T f7f06eded6c340c58101fcc644ebcf50 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.150s user 0.016s sys 0.020s
I20251214 20:52:38.257452 3221 raft_consensus.cc:515] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.258109 3221 leader_election.cc:290] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.258018 1206 raft_consensus.cc:359] T f7f06eded6c340c58101fcc644ebcf50 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.258371 1206 raft_consensus.cc:385] T f7f06eded6c340c58101fcc644ebcf50 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.258476 1206 raft_consensus.cc:740] T f7f06eded6c340c58101fcc644ebcf50 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.258714 1206 consensus_queue.cc:260] T f7f06eded6c340c58101fcc644ebcf50 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.259456 1206 ts_tablet_manager.cc:1434] T f7f06eded6c340c58101fcc644ebcf50 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:38.260049 1206 tablet_bootstrap.cc:492] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:38.261792 1206 tablet_bootstrap.cc:654] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.267006 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25180d5352204cdca3c49f907a587cb2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.268712 768 raft_consensus.cc:3060] T 054cfd587d19409fbdf7f4848bcaf3f9 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.274538 824 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:38.279347 1203 tablet_bootstrap.cc:492] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.287896 3287 raft_consensus.cc:493] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.288137 3287 raft_consensus.cc:515] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:38.289058 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 48237.8us
I20251214 20:52:38.289477 3287 leader_election.cc:290] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.292703 3111 raft_consensus.cc:697] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.293138 3111 consensus_queue.cc:237] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.297122 3108 raft_consensus.cc:697] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.297435 3108 consensus_queue.cc:237] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.306878 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25180d5352204cdca3c49f907a587cb2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.307513 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b482043af42485eb4c73d6eb864a963" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.307868 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1def5f0de5ce4eb0bed36f626c4f0683" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.308025 1203 tablet_bootstrap.cc:654] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.309033 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "880a78b924574181bdd804d599f4f579" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.309288 1202 tablet_bootstrap.cc:492] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:38.309410 1202 ts_tablet_manager.cc:1403] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.006s
I20251214 20:52:38.310452 1202 raft_consensus.cc:359] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.310487 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1def5f0de5ce4eb0bed36f626c4f0683" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.310734 1202 raft_consensus.cc:385] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.310741 1049 raft_consensus.cc:3060] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.310853 1202 raft_consensus.cc:740] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.311188 1202 consensus_queue.cc:260] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.311764 1202 ts_tablet_manager.cc:1434] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:38.313479 1202 tablet_bootstrap.cc:492] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:38.314927 1049 raft_consensus.cc:2468] T 1def5f0de5ce4eb0bed36f626c4f0683 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.324656 1203 tablet_bootstrap.cc:492] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.324829 1203 ts_tablet_manager.cc:1403] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.007s
I20251214 20:52:38.326083 1203 raft_consensus.cc:359] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.326253 1203 raft_consensus.cc:385] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.326304 1203 raft_consensus.cc:740] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.326511 1203 consensus_queue.cc:260] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.326993 1203 ts_tablet_manager.cc:1434] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:38.327283 1203 tablet_bootstrap.cc:492] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.327206 772 raft_consensus.cc:3060] T 8db4364b5023438bb43538855007c95e P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:38.331454 3066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20251214 20:52:38.332257 1203 tablet_bootstrap.cc:654] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:38.334771 3066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251214 20:52:38.335006 3066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251214 20:52:38.344471 772 raft_consensus.cc:3060] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.345126 1203 tablet_bootstrap.cc:492] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.345294 1203 ts_tablet_manager.cc:1403] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251214 20:52:38.346082 1203 raft_consensus.cc:359] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.346258 1203 raft_consensus.cc:385] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.346318 1203 raft_consensus.cc:740] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.346552 1203 consensus_queue.cc:260] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.347026 1203 ts_tablet_manager.cc:1434] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:38.347961 1203 tablet_bootstrap.cc:492] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.350025 1203 tablet_bootstrap.cc:654] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.352231 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25180d5352204cdca3c49f907a587cb2" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.352456 1053 raft_consensus.cc:3060] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.355597 1053 raft_consensus.cc:2468] T 25180d5352204cdca3c49f907a587cb2 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.356376 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b482043af42485eb4c73d6eb864a963" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.356576 1049 raft_consensus.cc:3060] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.361094 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "880a78b924574181bdd804d599f4f579" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.361358 1056 raft_consensus.cc:2468] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:38.362329 848 leader_election.cc:343] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.363452 848 leader_election.cc:343] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.364806 848 leader_election.cc:343] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.365321 848 leader_election.cc:343] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.365787 848 leader_election.cc:343] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.366197 848 leader_election.cc:343] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.368821 848 leader_election.cc:343] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.368853 1202 tablet_bootstrap.cc:654] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.370378 1203 tablet_bootstrap.cc:492] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.370519 1203 ts_tablet_manager.cc:1403] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20251214 20:52:38.371305 1203 raft_consensus.cc:359] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.371469 1203 raft_consensus.cc:385] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.371519 1203 raft_consensus.cc:740] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.371680 1203 consensus_queue.cc:260] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.373524 1203 ts_tablet_manager.cc:1434] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:38.373852 1203 tablet_bootstrap.cc:492] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.376222 1203 tablet_bootstrap.cc:654] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.378130 1049 raft_consensus.cc:2468] T 6b482043af42485eb4c73d6eb864a963 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.380959 851 leader_election.cc:304] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.381333 851 leader_election.cc:304] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.381665 3111 raft_consensus.cc:2804] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.381767 3111 raft_consensus.cc:697] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.381913 3108 raft_consensus.cc:2804] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.381986 3111 consensus_queue.cc:237] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.384204 3453 consensus_queue.cc:1048] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:38.384682 3066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251214 20:52:38.384883 3066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251214 20:52:38.385051 3066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251214 20:52:38.385171 3066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251214 20:52:38.385385 851 leader_election.cc:304] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.385881 3453 raft_consensus.cc:2804] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.385952 3453 raft_consensus.cc:493] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.386013 3453 raft_consensus.cc:3060] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.388612 3453 raft_consensus.cc:515] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.389083 1206 tablet_bootstrap.cc:492] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:38.389249 1206 ts_tablet_manager.cc:1403] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.129s user 0.034s sys 0.018s
I20251214 20:52:38.389595 3453 leader_election.cc:290] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
W20251214 20:52:38.390033 848 leader_election.cc:343] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.390084 1206 raft_consensus.cc:359] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.390344 1206 raft_consensus.cc:385] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.390429 1206 raft_consensus.cc:740] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.390712 1206 consensus_queue.cc:260] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:38.391496 848 consensus_peers.cc:597] T 49778cfe80f6483389cfd421abee8ca7 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:38.392428 848 leader_election.cc:343] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.392771 848 leader_election.cc:343] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.392830 1203 tablet_bootstrap.cc:492] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.392954 1203 ts_tablet_manager.cc:1403] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
W20251214 20:52:38.392977 848 leader_election.cc:343] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.393230 848 leader_election.cc:343] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.393718 851 leader_election.cc:304] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.393746 1203 raft_consensus.cc:359] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.393925 1203 raft_consensus.cc:385] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.393992 1203 raft_consensus.cc:740] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.394239 1203 consensus_queue.cc:260] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.395215 1203 ts_tablet_manager.cc:1434] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:38.395920 3452 raft_consensus.cc:2804] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.396086 3452 raft_consensus.cc:697] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.396371 3452 consensus_queue.cc:237] T 6b482043af42485eb4c73d6eb864a963 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.396559 1206 ts_tablet_manager.cc:1434] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251214 20:52:38.396896 1206 tablet_bootstrap.cc:492] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:38.398852 1206 tablet_bootstrap.cc:654] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.402518 2906 raft_consensus.cc:493] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.402714 2906 raft_consensus.cc:515] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.403451 2906 leader_election.cc:290] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.404150 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "248e3666fd13434e97378eb5a98dfadd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:38.404893 991 leader_election.cc:343] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.405673 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "248e3666fd13434e97378eb5a98dfadd" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.406322 989 leader_election.cc:343] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.406445 989 leader_election.cc:304] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.407020 2906 raft_consensus.cc:2749] T 248e3666fd13434e97378eb5a98dfadd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.411365 3479 tablet.cc:1722] T 4501fd5566a34e2a82009cc9ce6bdd37 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:38.413209 1203 tablet_bootstrap.cc:492] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.415899 1203 tablet_bootstrap.cc:654] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.416186 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "880a78b924574181bdd804d599f4f579" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.412832 1206 tablet_bootstrap.cc:492] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:38.417126 1206 ts_tablet_manager.cc:1403] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20251214 20:52:38.417971 1206 raft_consensus.cc:359] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.418161 1206 raft_consensus.cc:385] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.418218 1206 raft_consensus.cc:740] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.418416 1206 consensus_queue.cc:260] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.418918 1206 ts_tablet_manager.cc:1434] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:38.423350 1206 tablet_bootstrap.cc:492] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:38.425251 1206 tablet_bootstrap.cc:654] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.382097 3108 raft_consensus.cc:697] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.427716 3108 consensus_queue.cc:237] T 25180d5352204cdca3c49f907a587cb2 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.429388 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet ba322cb6ee084bc39788ddd439f5f15d (table default.Table_196d0f488e0b42aabd518511a4289dc3 [id=2f57c28c9c624a6ca1986c8831b85169]). Expected version 1 got 0
I20251214 20:52:38.434835 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "880a78b924574181bdd804d599f4f579" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.435087 1056 raft_consensus.cc:3060] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:38.435097 848 leader_election.cc:343] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.438346 1056 raft_consensus.cc:2468] T 880a78b924574181bdd804d599f4f579 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.443868 2991 raft_consensus.cc:493] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.444083 2991 raft_consensus.cc:515] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.444706 2991 leader_election.cc:290] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:38.460096 713 leader_election.cc:343] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: e9f6da5d29294f08a86c96c6821dda10
I20251214 20:52:38.460675 851 leader_election.cc:304] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.461104 3451 raft_consensus.cc:2804] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.461216 3451 raft_consensus.cc:697] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.461490 3451 consensus_queue.cc:237] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.468489 3480 tablet.cc:1722] T 719d157ac8f74eb7911df05ab5e4a3d5 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:38.470535 2906 raft_consensus.cc:493] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.470686 2906 raft_consensus.cc:515] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.471315 2906 leader_election.cc:290] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.471993 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b87b0871c2454971a15dd5643c155694" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:38.472733 991 leader_election.cc:343] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.474571 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b87b0871c2454971a15dd5643c155694" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.475203 989 leader_election.cc:343] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.475302 989 leader_election.cc:304] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.475697 2906 raft_consensus.cc:2749] T b87b0871c2454971a15dd5643c155694 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.487512 3488 tablet.cc:1722] T 719d157ac8f74eb7911df05ab5e4a3d5 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:38.491281 1203 tablet_bootstrap.cc:492] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.491488 1203 ts_tablet_manager.cc:1403] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.005s
I20251214 20:52:38.492388 1203 raft_consensus.cc:359] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.493391 1203 raft_consensus.cc:385] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.493489 1203 raft_consensus.cc:740] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.497032 774 raft_consensus.cc:3060] T 4dcd511a5dca4a2cbda81fbc255ce354 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.501013 1202 tablet_bootstrap.cc:492] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:38.501183 1202 ts_tablet_manager.cc:1403] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.188s user 0.007s sys 0.005s
I20251214 20:52:38.502002 1202 raft_consensus.cc:359] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.502197 1202 raft_consensus.cc:385] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.502261 1202 raft_consensus.cc:740] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.502467 1202 consensus_queue.cc:260] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:38.503930 848 consensus_peers.cc:597] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:38.505820 3287 consensus_queue.cc:1048] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:38.514569 3485 tablet.cc:1722] T 719d157ac8f74eb7911df05ab5e4a3d5 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251214 20:52:38.515493 1202 ts_tablet_manager.cc:1434] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251214 20:52:38.516256 1202 tablet_bootstrap.cc:492] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:38.518275 1202 tablet_bootstrap.cc:654] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:38.518230 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:38123, user_credentials={real_user=slave}} blocked reactor thread for 77734.5us
I20251214 20:52:38.519057 2906 raft_consensus.cc:493] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.519240 2906 raft_consensus.cc:515] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.519886 2906 leader_election.cc:290] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.521865 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52c05204502a45e18c2b862e21c6f361" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.522251 3371 raft_consensus.cc:493] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.522384 3371 raft_consensus.cc:515] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.522926 3371 leader_election.cc:290] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:38.523952 991 leader_election.cc:343] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.524335 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "197d3d340f6e4cbea0d621907b8c76d4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.525286 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "197d3d340f6e4cbea0d621907b8c76d4" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.526010 991 leader_election.cc:343] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.526286 989 leader_election.cc:343] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.526369 989 leader_election.cc:304] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.526805 3371 raft_consensus.cc:2749] T 197d3d340f6e4cbea0d621907b8c76d4 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.527772 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "52c05204502a45e18c2b862e21c6f361" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.528362 989 leader_election.cc:343] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.528465 989 leader_election.cc:304] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.528880 3371 raft_consensus.cc:2749] T 52c05204502a45e18c2b862e21c6f361 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.535291 1203 consensus_queue.cc:260] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.537318 1203 ts_tablet_manager.cc:1434] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20251214 20:52:38.544704 1203 tablet_bootstrap.cc:492] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.546681 3452 raft_consensus.cc:493] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.546852 3452 raft_consensus.cc:515] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.547482 3452 leader_election.cc:290] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.547850 3371 raft_consensus.cc:493] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.548008 3371 raft_consensus.cc:515] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.548656 3371 leader_election.cc:290] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.549474 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba548574d8784701974e611d7a9ad9b1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:38.550137 991 leader_election.cc:343] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.551321 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7b247b761c94c60ad3a468db371b30b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.551586 1049 raft_consensus.cc:2468] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.552278 851 leader_election.cc:304] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.552356 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba548574d8784701974e611d7a9ad9b1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.553022 989 leader_election.cc:343] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.553081 3287 raft_consensus.cc:2804] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.553154 3287 raft_consensus.cc:493] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.553148 989 leader_election.cc:304] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.553207 3287 raft_consensus.cc:3060] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.553683 3371 raft_consensus.cc:2749] T ba548574d8784701974e611d7a9ad9b1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.557516 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7b247b761c94c60ad3a468db371b30b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.559008 712 leader_election.cc:343] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: e9f6da5d29294f08a86c96c6821dda10
W20251214 20:52:38.559734 848 leader_election.cc:343] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.561429 712 leader_election.cc:304] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.550881 2325 catalog_manager.cc:5654] T 9bf9e65040af4fbeb701619fe27edfb6 P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:38.562505 3272 raft_consensus.cc:2749] T e9f6da5d29294f08a86c96c6821dda10 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.562680 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 4b5360e7ece94fd2a5a6512db6c6c12f (table default.Table_2ca54bf356e24a1ebca6f4a155b83044 [id=7f407f5abea945498cf939a23b0f9319]). Expected version 1 got 0
I20251214 20:52:38.563993 3287 raft_consensus.cc:515] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.565884 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7b247b761c94c60ad3a468db371b30b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.566126 1056 raft_consensus.cc:3060] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.568041 3066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42865,127.0.104.61:34819,127.0.104.60:35099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=44579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type FASTDEBUG
built by None at 14 Dec 2025 20:43:23 UTC on 24a791456cd2
build id 9516
ASAN enabled
I20251214 20:52:38.569034 1056 raft_consensus.cc:2468] T e7b247b761c94c60ad3a468db371b30b P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.569677 851 leader_election.cc:304] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.570427 3452 raft_consensus.cc:2804] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.570554 3452 raft_consensus.cc:697] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.570791 3452 consensus_queue.cc:237] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.583691 3287 leader_election.cc:290] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.585053 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]). Expected version 1 got 0
I20251214 20:52:38.585453 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 76472ce9005b4bd2bbc51278816e8ae0 (table default.Table_9698b9206b934e4db2decdf9c0180838 [id=4b3f6c8d8cce4f7a96b2c677df6469ba]). Expected version 1 got 0
I20251214 20:52:38.585795 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_5de6830aaec645f2992e4cfaceb20c27 [id=9553b46856c94003a293135c814a7df7]). Expected version 2 got 0
I20251214 20:52:38.586813 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7b247b761c94c60ad3a468db371b30b" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:38.589455 848 leader_election.cc:343] T e7b247b761c94c60ad3a468db371b30b P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.589955 848 consensus_peers.cc:597] T bd1161eebe884fe6b347bf517269c047 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:38.592873 1203 tablet_bootstrap.cc:654] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.586081 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 80098b7c24824d129b8f25daf2115460 (table default.Table_b14d897359c140bab6d04403fd59f31a [id=2a055be9028342be97f5c98f03f5a137]). Expected version 2 got 0
I20251214 20:52:38.597689 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 8db4364b5023438bb43538855007c95e (table default.Table_80d7626996ca40979563d484ae097f08 [id=694ef81722d948638bbec8847c69d75a]). Expected version 1 got 0
I20251214 20:52:38.599494 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 59ccb6c3b35044dc913cce74b5802e38 (table default.Table_d61784eed0b84f8caa485dbe70cdc366 [id=b5622661ea9f40f7ac11f44ae92a5194]). Expected version 1 got 0
I20251214 20:52:38.601940 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet f11834cabdad445b8e53cd237d146282 (table default.Table_398f36b3acb440d88213e38f63740ca5 [id=954b3b4acdd94043b2608438d00483e7]). Expected version 1 got 0
I20251214 20:52:38.602275 2325 catalog_manager.cc:5654] T 2d5920fdba794e5f864c595e7be278ab P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:38.602572 1202 tablet_bootstrap.cc:492] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:38.602797 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 8a818ca230564b93ac165d43ff22deb1 (table default.Table_b289a25bf8174d9199c90212ce70271d [id=0a8ffa32e9ce4e3a8f9fa2afa4de69bb]). Expected version 1 got 0
I20251214 20:52:38.602821 1202 ts_tablet_manager.cc:1403] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.087s user 0.023s sys 0.003s
I20251214 20:52:38.603003 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 082ec70b8ee646b898d9201ad073d292 (table default.Table_e3feb6d6cacb48dca8455eacf4cfb84b [id=8f720514e13b45689817a9811fb624d9]). Expected version 1 got 0
I20251214 20:52:38.603421 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 016581519f4c4e0d832b49346d55b41c (table default.Table_531eb871a88242c5b91423a2c6abc3b6 [id=9afdb569356648bb84089b1738dfc359]). Expected version 1 got 0
I20251214 20:52:38.603579 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 96a2b444608e42d687eeba2dc0ebb757 (table default.Table_90e94bab31e54a9cb8b567622aa4e37d [id=7c3d3542270e449ca954d068bece3491]). Expected version 1 got 0
I20251214 20:52:38.603523 1202 raft_consensus.cc:359] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.603693 1202 raft_consensus.cc:385] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.603745 1202 raft_consensus.cc:740] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.603960 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (table default.Table_fcafcf3c6ddb4457aea41320f7bf0b74 [id=4329d01e7f6147d0adad53fa86369b17]). Expected version 1 got 0
I20251214 20:52:38.603957 1202 consensus_queue.cc:260] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.604732 1202 ts_tablet_manager.cc:1434] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:38.605324 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet a5d28f09ad174936b47debace0599600 (table default.Table_93730665b1ab4686a0d3e54740fbddc8 [id=0b5581be22d345b3a0fd22c3dce2f207]). Expected version 1 got 0
I20251214 20:52:38.605526 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet f6d51d07f7e9450a844be9afddb60d03 (table default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]). Expected version 1 got 0
I20251214 20:52:38.609062 1202 tablet_bootstrap.cc:492] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:38.611114 1202 tablet_bootstrap.cc:654] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:38.613528 2288 consensus_peers.cc:597] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad -> Peer c27224391721467a8f46e467a980b37a (127.0.104.62:42865): Couldn't send request to peer c27224391721467a8f46e467a980b37a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42865: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251214 20:52:38.613998 1206 tablet_bootstrap.cc:492] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:38.614090 3066 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251214 20:52:38.614265 1206 ts_tablet_manager.cc:1403] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.191s user 0.010s sys 0.036s
I20251214 20:52:38.615212 1206 raft_consensus.cc:359] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.615226 2325 catalog_manager.cc:5654] T cc8d725e6ac440899fc1cde2f7b546aa P a2e8fd1cb731446d88f8e510c7a52999 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "a2e8fd1cb731446d88f8e510c7a52999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } health_report { overall_health: HEALTHY } } }
I20251214 20:52:38.615520 1206 raft_consensus.cc:385] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.615598 1206 raft_consensus.cc:740] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.615885 1206 consensus_queue.cc:260] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.616158 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet cc8d725e6ac440899fc1cde2f7b546aa (table default.Table_1052ac11eee9483aa13db090acff5b7a [id=978e62482f114f2b99e9707aaa9a5e32]). Expected version 1 got 0
I20251214 20:52:38.616649 1206 ts_tablet_manager.cc:1434] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:38.616742 3066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251214 20:52:38.617154 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d2688515bd1441fbad7516f8343ca775 (table default.Table_f1e9c435093240e4b1c1eb95635c8e66 [id=03997e2eab154ef18b2e3774b5984764]). Expected version 2 got 0
I20251214 20:52:38.617620 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet a4643effa71d4d18850913a59ae1c75d (table default.Table_679d53b1472341ef8612ca53ac1ebda2 [id=91f3b4dd176348a5aa38c6bdc417918f]). Expected version 3 got 0
I20251214 20:52:38.618080 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:38.618347 3371 raft_consensus.cc:493] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.618417 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_72013f5e5d4141f0aadb221a11e76eb7 [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 3 got 0
I20251214 20:52:38.618508 3371 raft_consensus.cc:515] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.618602 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 0d0f7bd831b4474a90288ac7fa600436 (table default.Table_d8b86c6bc911454fac28b6384cf9ff3c [id=c97bcc9ac37c4413bae618ee1019e069]). Expected version 1 got 0
I20251214 20:52:38.618844 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:38.619289 3371 leader_election.cc:290] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.619441 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 69f9d5838e544002ab49f0ce21b6f7ef (table default.Table_b77f9ca5f0f84a4db069e0a93db601e9 [id=4a1816b57ff84e459394e59b683d8ec6]). Expected version 1 got 0
I20251214 20:52:38.619611 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738 (table default.Table_6d745e9b39424aecbb85b0afa871ccd5 [id=a56279b96c2d45af90277c7636192936]). Expected version 2 got 0
I20251214 20:52:38.620178 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15b3216f3d2b443e8ab2f994445956f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:38.621181 991 leader_election.cc:343] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.621517 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15b3216f3d2b443e8ab2f994445956f9" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.622314 989 leader_election.cc:343] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.622422 989 leader_election.cc:304] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.622951 3371 raft_consensus.cc:2749] T 15b3216f3d2b443e8ab2f994445956f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.623998 1206 tablet_bootstrap.cc:492] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:38.624014 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 202b1d53cab644c98838b0ee81f29f22 (table default.Table_cddd59a230e04595bffdf9e0a0ed1471 [id=6094596f6c064c5bb1e20aa2b8767b9f]). Expected version 2 got 0
I20251214 20:52:38.624200 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 9bd89709abbc49809a2cdb6d0c29fdb8 (table default.Table_c3aa8e3c31cb41dc8281962e7b5c46e1 [id=687eb544b4114bad8d2deb76388e2022]). Expected version 1 got 0
I20251214 20:52:38.624675 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d424e3387698434f92e84eb1d43f5949 (table default.Table_9098a11909c844dca6d4e80a284aa2e8 [id=1597e24e3fff4007ab66b44505c960d8]). Expected version 1 got 0
I20251214 20:52:38.624953 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 6302145a6e7b4297a58d9d7f3ad02dce (table default.Table_1be39fb576af4b2b9e2606b495619863 [id=54e0182cd3b9461aa593a11aa7709002]). Expected version 1 got 0
I20251214 20:52:38.625105 1203 tablet_bootstrap.cc:492] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.625267 1203 ts_tablet_manager.cc:1403] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.002s
I20251214 20:52:38.625771 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 526f28f9aaad445081d9d69cc20ed7b3 (table default.Table_b87980e2da6a4052b109938fcf0527d8 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 2 got 0
I20251214 20:52:38.626001 1203 raft_consensus.cc:359] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.626184 1203 raft_consensus.cc:385] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.626241 1203 raft_consensus.cc:740] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.626282 1206 tablet_bootstrap.cc:654] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.626454 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 438b8ec5e9834442994966c8d83c1a31 (table default.Table_5cffb2d16a4847caa5cf6a2c4c238da8 [id=a8b3bbe26fc14b4bbc8a78c3d1f9df6d]). Expected version 1 got 0
I20251214 20:52:38.626441 1203 consensus_queue.cc:260] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.626995 1203 ts_tablet_manager.cc:1434] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251214 20:52:38.627408 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet f891db00595b49ca9e6ae64721e4d744 (table default.Table_c404f3a3b71f4899a233f864e78a7d13 [id=203681807a4b40b0a1f69387cfecff38]). Expected version 1 got 0
I20251214 20:52:38.627636 1203 tablet_bootstrap.cc:492] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.627835 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet bea3e9d7334345fa88ededf1be17a434 (table default.Table_27579b7155b140b3bdda5014c1eb57a8 [id=e24a6c9e5c9049edba5e42e22d22a831]). Expected version 1 got 0
I20251214 20:52:38.628921 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet e0d0e8f35bf44c9a8797b7507dcce3b9 (table default.Table_4649d8b3e43c42a492723fadd4217e4e [id=75cebf8d0aba4aa48c1be51b0abd6b38]). Expected version 2 got 0
I20251214 20:52:38.629419 1203 tablet_bootstrap.cc:654] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.631839 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet e1dbaa39efa04086a22cc3908ed691d4 (table default.Table_8f897cb4591a4d5fadbf2b1b7ee7408d [id=4390cb960d564bb5bc7e0cefa5bf3004]). Expected version 1 got 0
I20251214 20:52:38.632261 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet d39e245bc5a04c1186d0973645aa1b54 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 4 got 0
I20251214 20:52:38.632702 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet f8e98027ac3f4f36aded3aa5b47e6cad (table default.Table_5c7e3a65102c45a1a23d29b95427fae5 [id=db0d9e33bf054daabddadd1248243609]). Expected version 1 got 0
I20251214 20:52:38.633044 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 780be07e6f5d425d9605402b6db63012 (table default.Table_0d865be331094b9d8c407969618d12d4 [id=b251a30885f6435fbfa430b6b50aee7a]). Expected version 1 got 0
I20251214 20:52:38.634577 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet a12e8053c65343758d518ee840dbaee8 (table default.Table_cb922bf98617480aa9363c3f08d918f3 [id=e815640c7ece4361800a22bc749915e9]). Expected version 1 got 0
I20251214 20:52:38.635243 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet c7885f21a369461c8c80ff18d369e2c6 (table default.Table_f9c22ae822b3401ab43ad74df0b0289c [id=a20045f442604c45a67b9b06f1eb4610]). Expected version 2 got 0
I20251214 20:52:38.635525 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 96157aae81624a4ba9fcf8c3f142911f (table default.Table_adb418bf04534b868c044988fb97e906 [id=d7e403a944e94d0d8c9d6ba115723ea4]). Expected version 1 got 0
W20251214 20:52:38.636646 848 consensus_peers.cc:597] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:38.637200 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 2398f176ee044c15841eaa40cb74974c (table default.Table_25630f29e4904aaf86f8f65078bcf81e [id=b1ee89b437fb42a4b5e2c3d386ee0284]). Expected version 1 got 0
I20251214 20:52:38.637450 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 2807af0754914688bd7a090b1bc32eb1 (table default.Table_c6c0496d455a4db68c05c0ff8b06493d [id=9d1473df58ee486aa6a55893aca9d54c]). Expected version 3 got 0
I20251214 20:52:38.637768 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet db0ad1aa4193467db74cd707e7084b71 (table default.Table_5c7eb24742b84515af287c257637e51b [id=8acc06886693413682b246f25970fa5b]). Expected version 1 got 0
I20251214 20:52:38.638087 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 5ec7824f5eff4cc59c45acd3b3b3d194 (table default.Table_6cfa43d70d2f426fb58f8ef32476c3ac [id=be1be47d388347e3a6b7a80105688be3]). Expected version 1 got 0
I20251214 20:52:38.644443 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 639cf3b7654a4ae896c6edd8b2ff1844 (table default.Table_15c1781afe3449ef87ef3c655ccca0df [id=02c00037bc944c00b0124d72af62e19b]). Expected version 1 got 0
I20251214 20:52:38.644888 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 894e3f27b7904db994711aa80a9b1600 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
I20251214 20:52:38.645300 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 3a0f669313244e809c2868ca0ccfad65 (table default.Table_d59e2b0d82384da68c1901ddaa419d00 [id=0c56f14b40ac442994e232aa348d64a2]). Expected version 1 got 0
I20251214 20:52:38.645998 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 0dbfe8ee68bf4a2c88489b1f81b36991 (table default.Table_53bf327c948f4cebbe6096266ac301f8 [id=6a279e57008549c593df838211cc9cc1]). Expected version 1 got 0
I20251214 20:52:38.646273 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 6e005feeb95b48b9ad33612ab81d5bce (table default.Table_ec8c8f9185be4470a6580c4e141e2227 [id=8872bfac95c64cbba1ccb84d1d9f2483]). Expected version 2 got 0
I20251214 20:52:38.646529 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet b87b0871c2454971a15dd5643c155694 (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:38.646471 3287 raft_consensus.cc:493] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.646647 3287 raft_consensus.cc:515] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.647301 3287 leader_election.cc:290] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.647964 2325 catalog_manager.cc:5746] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123) does not have the latest schema for tablet 67104e34df294ef5986171eca79cdc72 (table default.Table_27222912e9214b879d8a30975aba5688 [id=c0729ac770394d7b87ef36e53f85c151]). Expected version 3 got 0
I20251214 20:52:38.648530 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c62ffb71fff43d4812eb86f40ce1c58" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.635391 3272 raft_consensus.cc:493] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.648748 1056 raft_consensus.cc:2468] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.648934 3272 raft_consensus.cc:515] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.649724 3272 leader_election.cc:290] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.650014 3287 consensus_queue.cc:1048] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:38.650332 3108 consensus_queue.cc:1048] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:38.651306 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2688515bd1441fbad7516f8343ca775" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.652323 3108 raft_consensus.cc:493] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.652560 3108 raft_consensus.cc:515] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.653028 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2688515bd1441fbad7516f8343ca775" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.653357 3108 leader_election.cc:290] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.654955 851 leader_election.cc:304] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.655766 2777 raft_consensus.cc:2804] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.655866 2777 raft_consensus.cc:493] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.655925 2777 raft_consensus.cc:3060] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.659489 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c07966670d294a0495ec54d48c7f09cd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.659540 713 leader_election.cc:304] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.659730 1048 raft_consensus.cc:2468] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.660697 3272 raft_consensus.cc:2749] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.659063 2777 raft_consensus.cc:515] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.662628 851 leader_election.cc:304] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.662755 2777 leader_election.cc:290] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.663028 3108 raft_consensus.cc:2804] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.663403 3108 raft_consensus.cc:493] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.663513 3108 raft_consensus.cc:3060] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.663820 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c62ffb71fff43d4812eb86f40ce1c58" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.664032 1048 raft_consensus.cc:3060] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.664095 1203 tablet_bootstrap.cc:492] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.664260 1203 ts_tablet_manager.cc:1403] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.007s
I20251214 20:52:38.665195 1203 raft_consensus.cc:359] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.665396 1203 raft_consensus.cc:385] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.665442 1203 raft_consensus.cc:740] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.665690 1203 consensus_queue.cc:260] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.666347 1203 ts_tablet_manager.cc:1434] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:38.666296 3108 raft_consensus.cc:515] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.666628 1203 tablet_bootstrap.cc:492] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.666743 1048 raft_consensus.cc:2468] T 0c62ffb71fff43d4812eb86f40ce1c58 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.668190 851 leader_election.cc:304] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.668555 1203 tablet_bootstrap.cc:654] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.668603 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c07966670d294a0495ec54d48c7f09cd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.668833 1049 raft_consensus.cc:3060] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.668854 3287 raft_consensus.cc:2804] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.668967 3287 raft_consensus.cc:697] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.669245 3287 consensus_queue.cc:237] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.673813 1049 raft_consensus.cc:2468] T c07966670d294a0495ec54d48c7f09cd P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.674290 3108 leader_election.cc:290] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.674490 851 leader_election.cc:304] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.675237 3108 raft_consensus.cc:2804] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.675226 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c62ffb71fff43d4812eb86f40ce1c58" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.675678 3108 raft_consensus.cc:697] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.676365 3108 consensus_queue.cc:237] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.676963 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c07966670d294a0495ec54d48c7f09cd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.678821 1203 tablet_bootstrap.cc:492] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.678985 1203 ts_tablet_manager.cc:1403] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20251214 20:52:38.680106 1203 raft_consensus.cc:359] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.680258 1203 raft_consensus.cc:385] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.680306 1203 raft_consensus.cc:740] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.680321 772 raft_consensus.cc:3060] T f7f06eded6c340c58101fcc644ebcf50 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.680522 1203 consensus_queue.cc:260] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.681083 1203 ts_tablet_manager.cc:1434] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251214 20:52:38.685669 848 leader_election.cc:343] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.686641 848 leader_election.cc:343] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.691586 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c07966670d294a0495ec54d48c7f09cd" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.694464 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c62ffb71fff43d4812eb86f40ce1c58" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.700927 1203 tablet_bootstrap.cc:492] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.702682 1203 tablet_bootstrap.cc:654] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:38.702667 848 leader_election.cc:343] T c07966670d294a0495ec54d48c7f09cd P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.703195 848 consensus_peers.cc:597] T 147c985fa7c94f7a9af57917bf17b29f P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:38.704851 848 leader_election.cc:343] T 0c62ffb71fff43d4812eb86f40ce1c58 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.718765 3287 consensus_queue.cc:1048] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:38.721186 772 raft_consensus.cc:3060] T d2688515bd1441fbad7516f8343ca775 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.722595 1203 tablet_bootstrap.cc:492] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.722738 1203 ts_tablet_manager.cc:1403] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.003s
I20251214 20:52:38.723486 1203 raft_consensus.cc:359] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.723651 1203 raft_consensus.cc:385] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.723701 1203 raft_consensus.cc:740] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.723923 1203 consensus_queue.cc:260] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.724462 1203 ts_tablet_manager.cc:1434] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:38.724735 1203 tablet_bootstrap.cc:492] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.726066 1203 tablet_bootstrap.cc:654] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.733685 1202 tablet_bootstrap.cc:492] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:38.735045 1202 ts_tablet_manager.cc:1403] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.008s
I20251214 20:52:38.736280 1202 raft_consensus.cc:359] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.736630 1202 raft_consensus.cc:385] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.736747 1202 raft_consensus.cc:740] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.737084 1202 consensus_queue.cc:260] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.738740 1202 ts_tablet_manager.cc:1434] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251214 20:52:38.739394 1202 tablet_bootstrap.cc:492] T 7242be189ca54c9182181e33d6743e6b P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:38.742393 1202 tablet_bootstrap.cc:654] T 7242be189ca54c9182181e33d6743e6b P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.745388 1203 tablet_bootstrap.cc:492] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.745563 1203 ts_tablet_manager.cc:1403] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.005s
I20251214 20:52:38.746373 1203 raft_consensus.cc:359] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.746567 1203 raft_consensus.cc:385] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.746623 1203 raft_consensus.cc:740] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.746855 1203 consensus_queue.cc:260] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.747539 1203 ts_tablet_manager.cc:1434] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251214 20:52:38.749243 1203 tablet_bootstrap.cc:492] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.751435 1203 tablet_bootstrap.cc:654] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:38.768733 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:38.769155 3371 raft_consensus.cc:493] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.769361 3371 raft_consensus.cc:515] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.770097 3371 leader_election.cc:290] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.770916 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639cf3b7654a4ae896c6edd8b2ff1844" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.771653 989 leader_election.cc:343] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.752170 848 consensus_peers.cc:597] T 812642cfcff940f899ef9d3644909ec4 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:38.770706 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "639cf3b7654a4ae896c6edd8b2ff1844" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:38.774307 991 leader_election.cc:343] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.774447 991 leader_election.cc:304] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.774842 3371 raft_consensus.cc:2749] T 639cf3b7654a4ae896c6edd8b2ff1844 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.775565 1206 tablet_bootstrap.cc:492] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:38.775839 1206 ts_tablet_manager.cc:1403] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.152s user 0.005s sys 0.002s
I20251214 20:52:38.776881 1206 raft_consensus.cc:359] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.777638 1206 raft_consensus.cc:385] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.777772 1206 raft_consensus.cc:740] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.778115 1206 consensus_queue.cc:260] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.778976 1206 ts_tablet_manager.cc:1434] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251214 20:52:38.780784 1206 tablet_bootstrap.cc:492] T 5157ef5f1fd541efbbd2887bf4505254 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
W20251214 20:52:38.781276 3515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:38.782687 3287 raft_consensus.cc:493] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.782846 3287 raft_consensus.cc:515] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.783315 1206 tablet_bootstrap.cc:654] T 5157ef5f1fd541efbbd2887bf4505254 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.783459 3287 leader_election.cc:290] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.784860 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41effce1ac504f27b619a4d9950202f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.785248 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41effce1ac504f27b619a4d9950202f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.785741 1049 raft_consensus.cc:2468] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.786603 851 leader_election.cc:304] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.787103 3451 raft_consensus.cc:2804] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.787212 3451 raft_consensus.cc:493] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.787268 3451 raft_consensus.cc:3060] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.772225 3108 raft_consensus.cc:493] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.789134 3108 raft_consensus.cc:515] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.789779 3108 leader_election.cc:290] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.790177 3486 raft_consensus.cc:493] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.790313 3486 raft_consensus.cc:515] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.790948 3486 leader_election.cc:290] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:38.791896 3451 raft_consensus.cc:515] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.792532 3451 leader_election.cc:290] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.794109 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e59b80ee919b4e1393fe317ed0e3a3d2" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.794544 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e59b80ee919b4e1393fe317ed0e3a3d2" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.796077 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41effce1ac504f27b619a4d9950202f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.796252 1056 raft_consensus.cc:3060] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.799458 1056 raft_consensus.cc:2468] T 41effce1ac504f27b619a4d9950202f9 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.800185 712 leader_election.cc:304] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.800833 3486 raft_consensus.cc:2749] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.801301 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 082ec70b8ee646b898d9201ad073d292 (table default.Table_e3feb6d6cacb48dca8455eacf4cfb84b [id=8f720514e13b45689817a9811fb624d9]). Expected version 1 got 0
I20251214 20:52:38.801887 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 016581519f4c4e0d832b49346d55b41c (table default.Table_531eb871a88242c5b91423a2c6abc3b6 [id=9afdb569356648bb84089b1738dfc359]). Expected version 1 got 0
I20251214 20:52:38.801910 851 leader_election.cc:304] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters:
I20251214 20:52:38.802434 3108 raft_consensus.cc:2804] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.802630 3108 raft_consensus.cc:697] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.802965 3108 consensus_queue.cc:237] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:38.803294 848 leader_election.cc:343] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.803750 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "185049f1ef43499aa1d15d99549255f7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.804009 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "185049f1ef43499aa1d15d99549255f7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.804236 1049 raft_consensus.cc:2468] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.804548 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41effce1ac504f27b619a4d9950202f9" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.805161 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 96a2b444608e42d687eeba2dc0ebb757 (table default.Table_90e94bab31e54a9cb8b567622aa4e37d [id=7c3d3542270e449ca954d068bece3491]). Expected version 1 got 0
W20251214 20:52:38.805361 848 leader_election.cc:343] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.805545 848 leader_election.cc:343] T 41effce1ac504f27b619a4d9950202f9 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.806111 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet e6bdfbe4d15345de8be5de00bfe4ca07 (table default.Table_5de6830aaec645f2992e4cfaceb20c27 [id=9553b46856c94003a293135c814a7df7]). Expected version 2 got 0
I20251214 20:52:38.806411 851 leader_election.cc:304] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.806847 1206 tablet_bootstrap.cc:492] T 5157ef5f1fd541efbbd2887bf4505254 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:38.806984 3111 raft_consensus.cc:2804] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.807078 3111 raft_consensus.cc:493] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.807158 3111 raft_consensus.cc:3060] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.807204 1206 ts_tablet_manager.cc:1403] T 5157ef5f1fd541efbbd2887bf4505254 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.003s
I20251214 20:52:38.807197 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 4501fd5566a34e2a82009cc9ce6bdd37 (table default.Table_dce91ff3d9c74c7db1d400a24c4e550a [id=14f12b293a274ebeaf5bfefdff5b78cf]). Expected version 1 got 0
I20251214 20:52:38.807446 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 876d2a1f1d1c4f6882a5857b43568ee5 (table default.Table_fcafcf3c6ddb4457aea41320f7bf0b74 [id=4329d01e7f6147d0adad53fa86369b17]). Expected version 1 got 0
I20251214 20:52:38.807698 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet a5d28f09ad174936b47debace0599600 (table default.Table_93730665b1ab4686a0d3e54740fbddc8 [id=0b5581be22d345b3a0fd22c3dce2f207]). Expected version 1 got 0
I20251214 20:52:38.807861 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet f6d51d07f7e9450a844be9afddb60d03 (table default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3]). Expected version 1 got 0
I20251214 20:52:38.808010 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet a4643effa71d4d18850913a59ae1c75d (table default.Table_679d53b1472341ef8612ca53ac1ebda2 [id=91f3b4dd176348a5aa38c6bdc417918f]). Expected version 3 got 0
I20251214 20:52:38.808207 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 658e4fe0d188483b9ceb9eb3b3f20835 (table default.Table_ae63b32352284405b5918f5b33f80e5f [id=52a7552cd4b44ca68f0ad5f1538a99a7]). Expected version 1 got 0
I20251214 20:52:38.809206 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 69f9d5838e544002ab49f0ce21b6f7ef (table default.Table_b77f9ca5f0f84a4db069e0a93db601e9 [id=4a1816b57ff84e459394e59b683d8ec6]). Expected version 1 got 0
I20251214 20:52:38.809777 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 7c2a6c6c0b6e46bcb7658fbeced6f738 (table default.Table_6d745e9b39424aecbb85b0afa871ccd5 [id=a56279b96c2d45af90277c7636192936]). Expected version 2 got 0
I20251214 20:52:38.809793 3111 raft_consensus.cc:515] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.809967 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 202b1d53cab644c98838b0ee81f29f22 (table default.Table_cddd59a230e04595bffdf9e0a0ed1471 [id=6094596f6c064c5bb1e20aa2b8767b9f]). Expected version 2 got 0
I20251214 20:52:38.810115 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d424e3387698434f92e84eb1d43f5949 (table default.Table_9098a11909c844dca6d4e80a284aa2e8 [id=1597e24e3fff4007ab66b44505c960d8]). Expected version 1 got 0
I20251214 20:52:38.810283 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 9bd89709abbc49809a2cdb6d0c29fdb8 (table default.Table_c3aa8e3c31cb41dc8281962e7b5c46e1 [id=687eb544b4114bad8d2deb76388e2022]). Expected version 1 got 0
I20251214 20:52:38.810384 3111 leader_election.cc:290] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.810600 1206 raft_consensus.cc:359] T 5157ef5f1fd541efbbd2887bf4505254 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.810763 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 67104e34df294ef5986171eca79cdc72 (table default.Table_27222912e9214b879d8a30975aba5688 [id=c0729ac770394d7b87ef36e53f85c151]). Expected version 3 got 0
I20251214 20:52:38.810798 1206 raft_consensus.cc:385] T 5157ef5f1fd541efbbd2887bf4505254 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.810849 1206 raft_consensus.cc:740] T 5157ef5f1fd541efbbd2887bf4505254 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.812283 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "185049f1ef43499aa1d15d99549255f7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.812472 1053 raft_consensus.cc:3060] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.812992 1206 consensus_queue.cc:260] T 5157ef5f1fd541efbbd2887bf4505254 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.816350 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "185049f1ef43499aa1d15d99549255f7" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.816402 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 526f28f9aaad445081d9d69cc20ed7b3 (table default.Table_b87980e2da6a4052b109938fcf0527d8 [id=88d5807cc76646e3a2962b5d36413894]). Expected version 2 got 0
I20251214 20:52:38.816601 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 4dcd511a5dca4a2cbda81fbc255ce354 (table default.Table_03ec8d3fa1ea4120b178f12195e8c38a [id=5095fbf5b9014d708cf7e7f48a4be93e]). Expected version 1 got 0
I20251214 20:52:38.816756 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 719d157ac8f74eb7911df05ab5e4a3d5 (table default.Table_72013f5e5d4141f0aadb221a11e76eb7 [id=b0f347532cc84fcca5892f5a2443321c]). Expected version 3 got 0
I20251214 20:52:38.816967 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 0d0f7bd831b4474a90288ac7fa600436 (table default.Table_d8b86c6bc911454fac28b6384cf9ff3c [id=c97bcc9ac37c4413bae618ee1019e069]). Expected version 1 got 0
I20251214 20:52:38.817238 1206 ts_tablet_manager.cc:1434] T 5157ef5f1fd541efbbd2887bf4505254 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251214 20:52:38.818637 1206 tablet_bootstrap.cc:492] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:38.817328 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet f11834cabdad445b8e53cd237d146282 (table default.Table_398f36b3acb440d88213e38f63740ca5 [id=954b3b4acdd94043b2608438d00483e7]). Expected version 1 got 0
I20251214 20:52:38.819834 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8a818ca230564b93ac165d43ff22deb1 (table default.Table_b289a25bf8174d9199c90212ce70271d [id=0a8ffa32e9ce4e3a8f9fa2afa4de69bb]). Expected version 1 got 0
W20251214 20:52:38.820127 848 leader_election.cc:343] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.820778 1053 raft_consensus.cc:2468] T 185049f1ef43499aa1d15d99549255f7 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.820602 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet b87b0871c2454971a15dd5643c155694 (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749]). Expected version 1 got 0
I20251214 20:52:38.821475 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 161fa219cc2f4c44bbcfbdfe9dce9a1f (table default.Table_ab93057983a146d488d93fb5b803be4f [id=fc5b05a963ff4c40a975fb895072265e]). Expected version 1 got 0
I20251214 20:52:38.821993 851 leader_election.cc:304] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.822260 3453 raft_consensus.cc:2804] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.822358 3453 raft_consensus.cc:697] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.822425 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet cc8d725e6ac440899fc1cde2f7b546aa (table default.Table_1052ac11eee9483aa13db090acff5b7a [id=978e62482f114f2b99e9707aaa9a5e32]). Expected version 1 got 0
I20251214 20:52:38.822604 3453 consensus_queue.cc:237] T 185049f1ef43499aa1d15d99549255f7 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.822767 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 6302145a6e7b4297a58d9d7f3ad02dce (table default.Table_1be39fb576af4b2b9e2606b495619863 [id=54e0182cd3b9461aa593a11aa7709002]). Expected version 1 got 0
I20251214 20:52:38.823426 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 438b8ec5e9834442994966c8d83c1a31 (table default.Table_5cffb2d16a4847caa5cf6a2c4c238da8 [id=a8b3bbe26fc14b4bbc8a78c3d1f9df6d]). Expected version 1 got 0
I20251214 20:52:38.825743 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet f8e98027ac3f4f36aded3aa5b47e6cad (table default.Table_5c7e3a65102c45a1a23d29b95427fae5 [id=db0d9e33bf054daabddadd1248243609]). Expected version 1 got 0
I20251214 20:52:38.827047 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet bea3e9d7334345fa88ededf1be17a434 (table default.Table_27579b7155b140b3bdda5014c1eb57a8 [id=e24a6c9e5c9049edba5e42e22d22a831]). Expected version 1 got 0
I20251214 20:52:38.828533 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet f891db00595b49ca9e6ae64721e4d744 (table default.Table_c404f3a3b71f4899a233f864e78a7d13 [id=203681807a4b40b0a1f69387cfecff38]). Expected version 1 got 0
I20251214 20:52:38.828464 2777 consensus_queue.cc:1048] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:38.829810 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet e0d0e8f35bf44c9a8797b7507dcce3b9 (table default.Table_4649d8b3e43c42a492723fadd4217e4e [id=75cebf8d0aba4aa48c1be51b0abd6b38]). Expected version 2 got 0
I20251214 20:52:38.829834 1206 tablet_bootstrap.cc:654] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.835309 3371 raft_consensus.cc:493] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.835487 3371 raft_consensus.cc:515] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.836113 3371 leader_election.cc:290] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.837101 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2807af0754914688bd7a090b1bc32eb1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:38.837656 991 leader_election.cc:343] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.837920 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2807af0754914688bd7a090b1bc32eb1" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.838380 989 leader_election.cc:343] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.838495 989 leader_election.cc:304] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.838913 3371 raft_consensus.cc:2749] T 2807af0754914688bd7a090b1bc32eb1 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.841753 3451 raft_consensus.cc:493] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.841900 3451 raft_consensus.cc:515] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.842496 3451 leader_election.cc:290] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:38.843289 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d535d8f71b44c81bbba6937e804d900" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.843713 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d535d8f71b44c81bbba6937e804d900" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.843891 1053 raft_consensus.cc:2468] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
W20251214 20:52:38.843930 848 leader_election.cc:343] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.844272 851 leader_election.cc:304] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.845438 3451 raft_consensus.cc:2804] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.845526 3451 raft_consensus.cc:493] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.845599 3451 raft_consensus.cc:3060] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.845932 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet e1dbaa39efa04086a22cc3908ed691d4 (table default.Table_8f897cb4591a4d5fadbf2b1b7ee7408d [id=4390cb960d564bb5bc7e0cefa5bf3004]). Expected version 1 got 0
I20251214 20:52:38.846673 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d39e245bc5a04c1186d0973645aa1b54 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7]). Expected version 4 got 0
I20251214 20:52:38.847471 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet a12e8053c65343758d518ee840dbaee8 (table default.Table_cb922bf98617480aa9363c3f08d918f3 [id=e815640c7ece4361800a22bc749915e9]). Expected version 1 got 0
I20251214 20:52:38.850178 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2807af0754914688bd7a090b1bc32eb1 (table default.Table_c6c0496d455a4db68c05c0ff8b06493d [id=9d1473df58ee486aa6a55893aca9d54c]). Expected version 3 got 0
I20251214 20:52:38.852118 1206 tablet_bootstrap.cc:492] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:38.852797 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet c7885f21a369461c8c80ff18d369e2c6 (table default.Table_f9c22ae822b3401ab43ad74df0b0289c [id=a20045f442604c45a67b9b06f1eb4610]). Expected version 2 got 0
I20251214 20:52:38.854406 1206 ts_tablet_manager.cc:1403] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.000s
I20251214 20:52:38.855726 1206 raft_consensus.cc:359] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.856689 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 4b5360e7ece94fd2a5a6512db6c6c12f (table default.Table_2ca54bf356e24a1ebca6f4a155b83044 [id=7f407f5abea945498cf939a23b0f9319]). Expected version 1 got 0
I20251214 20:52:38.856751 3451 raft_consensus.cc:515] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.859639 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 780be07e6f5d425d9605402b6db63012 (table default.Table_0d865be331094b9d8c407969618d12d4 [id=b251a30885f6435fbfa430b6b50aee7a]). Expected version 1 got 0
I20251214 20:52:38.860271 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 96157aae81624a4ba9fcf8c3f142911f (table default.Table_adb418bf04534b868c044988fb97e906 [id=d7e403a944e94d0d8c9d6ba115723ea4]). Expected version 1 got 0
I20251214 20:52:38.858142 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d535d8f71b44c81bbba6937e804d900" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.859365 3039 raft_consensus.cc:493] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.860726 3039 raft_consensus.cc:515] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.861357 3039 leader_election.cc:290] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:38.861303 848 leader_election.cc:343] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.861654 1206 raft_consensus.cc:385] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.861778 1206 raft_consensus.cc:740] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.862528 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bed2c326906486e99300e05033c9c0f" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.862624 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 2398f176ee044c15841eaa40cb74974c (table default.Table_25630f29e4904aaf86f8f65078bcf81e [id=b1ee89b437fb42a4b5e2c3d386ee0284]). Expected version 1 got 0
I20251214 20:52:38.863206 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 5ec7824f5eff4cc59c45acd3b3b3d194 (table default.Table_6cfa43d70d2f426fb58f8ef32476c3ac [id=be1be47d388347e3a6b7a80105688be3]). Expected version 1 got 0
I20251214 20:52:38.863487 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet db0ad1aa4193467db74cd707e7084b71 (table default.Table_5c7eb24742b84515af287c257637e51b [id=8acc06886693413682b246f25970fa5b]). Expected version 1 got 0
I20251214 20:52:38.863816 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3bed2c326906486e99300e05033c9c0f" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.863971 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 639cf3b7654a4ae896c6edd8b2ff1844 (table default.Table_15c1781afe3449ef87ef3c655ccca0df [id=02c00037bc944c00b0124d72af62e19b]). Expected version 1 got 0
I20251214 20:52:38.864235 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 894e3f27b7904db994711aa80a9b1600 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc]). Expected version 2 got 0
I20251214 20:52:38.864488 712 leader_election.cc:304] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.864939 3039 raft_consensus.cc:2749] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.865002 1206 consensus_queue.cc:260] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.867655 1206 ts_tablet_manager.cc:1434] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20251214 20:52:38.872314 1206 tablet_bootstrap.cc:492] T cc8d725e6ac440899fc1cde2f7b546aa P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:38.872659 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d535d8f71b44c81bbba6937e804d900" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.872972 1053 raft_consensus.cc:3060] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.864537 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 3a0f669313244e809c2868ca0ccfad65 (table default.Table_d59e2b0d82384da68c1901ddaa419d00 [id=0c56f14b40ac442994e232aa348d64a2]). Expected version 1 got 0
I20251214 20:52:38.876286 1053 raft_consensus.cc:2468] T 6d535d8f71b44c81bbba6937e804d900 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.876374 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 0dbfe8ee68bf4a2c88489b1f81b36991 (table default.Table_53bf327c948f4cebbe6096266ac301f8 [id=6a279e57008549c593df838211cc9cc1]). Expected version 1 got 0
I20251214 20:52:38.876586 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 6e005feeb95b48b9ad33612ab81d5bce (table default.Table_ec8c8f9185be4470a6580c4e141e2227 [id=8872bfac95c64cbba1ccb84d1d9f2483]). Expected version 2 got 0
I20251214 20:52:38.877041 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet ba322cb6ee084bc39788ddd439f5f15d (table default.Table_196d0f488e0b42aabd518511a4289dc3 [id=2f57c28c9c624a6ca1986c8831b85169]). Expected version 1 got 0
I20251214 20:52:38.876993 851 leader_election.cc:304] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.877333 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 566c6e4761d5486d964e276d151e32b0 (table default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615]). Expected version 2 got 0
I20251214 20:52:38.877432 3287 raft_consensus.cc:2804] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:38.877497 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 8db4364b5023438bb43538855007c95e (table default.Table_80d7626996ca40979563d484ae097f08 [id=694ef81722d948638bbec8847c69d75a]). Expected version 1 got 0
I20251214 20:52:38.877549 3287 raft_consensus.cc:697] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:38.877779 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet d2688515bd1441fbad7516f8343ca775 (table default.Table_f1e9c435093240e4b1c1eb95635c8e66 [id=03997e2eab154ef18b2e3774b5984764]). Expected version 2 got 0
I20251214 20:52:38.877795 3287 consensus_queue.cc:237] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:38.877919 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 76472ce9005b4bd2bbc51278816e8ae0 (table default.Table_9698b9206b934e4db2decdf9c0180838 [id=4b3f6c8d8cce4f7a96b2c677df6469ba]). Expected version 1 got 0
I20251214 20:52:38.878139 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 59ccb6c3b35044dc913cce74b5802e38 (table default.Table_d61784eed0b84f8caa485dbe70cdc366 [id=b5622661ea9f40f7ac11f44ae92a5194]). Expected version 1 got 0
I20251214 20:52:38.878332 2328 catalog_manager.cc:5746] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835) does not have the latest schema for tablet 80098b7c24824d129b8f25daf2115460 (table default.Table_b14d897359c140bab6d04403fd59f31a [id=2a055be9028342be97f5c98f03f5a137]). Expected version 2 got 0
I20251214 20:52:38.878760 2321 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 76d16819e133434eb1c5a65fffb1f03f
I20251214 20:52:38.879510 2321 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41562 (request call id 73) took 1716 ms. Trace:
I20251214 20:52:38.879663 2321 rpcz_store.cc:276] 1214 20:52:37.162738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:37.338277 (+175539us) service_pool.cc:225] Handling call
1214 20:52:37.356377 (+ 18100us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x616000050854. stack: 00007fba1fa45ee6 00007fba1fa45ca0 00007fba1ea213ad 0000000000984f94 0000000000984f5d 00007fba2ed6b6bd 00007fba2ed6b7b3 00007fba2b932521 00007fba335f196d 00007fba3377f1b6 00007fba2a28a486 00007fba2a28a1b5 00007fba26a0456e 00007fba26a0344e 00007fba26a09685 00007fba26a09faf
1214 20:52:37.356886 (+ 509us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:37.357013 (+ 127us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:37.357108 (+ 95us) catalog_manager.cc:2824] Locking tablets
1214 20:52:37.357265 (+ 157us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:37.465915 (+108650us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:37.547303 (+ 81388us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:38.878676 (+1331373us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:38.879447 (+ 771us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:37.358246 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:37.358393 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:37.358401 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:37.358403 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:37.358535 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:37.358562 (+ 27us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:37.358566 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:37.358693 (+ 127us) tablet.cc:699] Row locks acquired
1214 20:52:37.358696 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:37.358745 (+ 49us) write_op.cc:270] Start()
1214 20:52:37.358792 (+ 47us) write_op.cc:276] Timestamp: P: 1765745557358694 usec, L: 0
1214 20:52:37.358796 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:37.359026 (+ 230us) log.cc:844] Serialized 756 byte log entry
1214 20:52:37.464216 (+105190us) op_driver.cc:464] REPLICATION: finished
1214 20:52:37.464414 (+ 198us) write_op.cc:301] APPLY: starting
1214 20:52:37.464436 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:37.464504 (+ 68us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:37.464533 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:37.464649 (+ 116us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:37.464690 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:37.464698 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:37.464807 (+ 109us) log.cc:844] Serialized 29 byte log entry
1214 20:52:37.464893 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:37.464919 (+ 26us) write_op.cc:454] Released schema lock
1214 20:52:37.464928 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:37.465013 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:37.465017 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":41014656,"child_traces":[["op",{"apply.queue_time_us":134,"apply.run_cpu_time_us":590,"apply.run_wall_time_us":590,"num_ops":2,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":859,"prepare.run_wall_time_us":860,"replication_time_us":105376}]]}
I20251214 20:52:38.883342 2322 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet d5e0b89d02c04625b8e31910d60d5d40
I20251214 20:52:38.875101 1206 tablet_bootstrap.cc:654] T cc8d725e6ac440899fc1cde2f7b546aa P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.884064 1203 tablet_bootstrap.cc:492] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.879995 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 76d16819e133434eb1c5a65fffb1f03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:38.884145 2322 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41562 (request call id 71) took 1721 ms. Trace:
I20251214 20:52:38.884229 1203 ts_tablet_manager.cc:1403] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.135s user 0.005s sys 0.004s
I20251214 20:52:38.884253 2322 rpcz_store.cc:276] 1214 20:52:37.162526 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:37.165166 (+ 2640us) service_pool.cc:225] Handling call
1214 20:52:37.280386 (+115220us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:37.280581 (+ 195us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:37.280698 (+ 117us) catalog_manager.cc:2824] Locking tablets
1214 20:52:37.280903 (+ 205us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:37.367209 (+ 86306us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:37.546713 (+179504us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:38.883261 (+1336548us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:38.884078 (+ 817us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:37.281962 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:37.282106 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:37.282114 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:37.282116 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:37.282243 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:37.282254 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:37.282258 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:37.282393 (+ 135us) tablet.cc:699] Row locks acquired
1214 20:52:37.282396 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:37.282433 (+ 37us) write_op.cc:270] Start()
1214 20:52:37.282472 (+ 39us) write_op.cc:276] Timestamp: P: 1765745557282403 usec, L: 0
1214 20:52:37.282475 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:37.282700 (+ 225us) log.cc:844] Serialized 725 byte log entry
1214 20:52:37.355832 (+ 73132us) spinlock_profiling.cc:243] Waited 21.5 ms on lock 0x60c0004d1b00. stack: 00007fba1fa45ee6 00007fba1fa45ca0 00007fba1ea213ad 0000000000984f94 0000000000984f5d 000000000098492a 00007fba33626cc5 00007fba2ccea764 00007fba2ccee29c 00007fba2ccedfc1 00007fba336e3efd 00007fba2b93cc7c 00007fba2b8c7979 00007fba2b908e27 00007fba2b84a0bc 00007fba2b849e9c
1214 20:52:37.355842 (+ 10us) op_driver.cc:464] REPLICATION: finished
1214 20:52:37.356112 (+ 270us) spinlock_profiling.cc:243] Waited 72.9 ms on lock 0x60c0004d1b00. stack: 00007fba1fa45ee6 00007fba1fa45ca0 00007fba1ea213ad 0000000000984f94 0000000000984f5d 000000000098492a 00007fba33626cc5 00007fba1fa8c09e 00007fba1fa8f52b 00007fba1fa8f3a8 0000000000a8970c 00007fba1fa6b3f8 00007fba230766da 00007fba1bba771e
1214 20:52:37.364523 (+ 8411us) write_op.cc:301] APPLY: starting
1214 20:52:37.364554 (+ 31us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:37.364630 (+ 76us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:37.364633 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:37.364731 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:37.364787 (+ 56us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:37.364795 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:37.364907 (+ 112us) log.cc:844] Serialized 29 byte log entry
1214 20:52:37.364991 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:37.365019 (+ 28us) write_op.cc:454] Released schema lock
1214 20:52:37.365028 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:37.365128 (+ 100us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:37.365132 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":8570,"apply.run_cpu_time_us":608,"apply.run_wall_time_us":608,"num_ops":2,"prepare.queue_time_us":76,"prepare.run_cpu_time_us":870,"prepare.run_wall_time_us":871,"raft.queue_time_us":261,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":51350,"spinlock_wait_cycles":217294592}]]}
I20251214 20:52:38.885098 1203 raft_consensus.cc:359] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.885290 1203 raft_consensus.cc:385] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.885337 1203 raft_consensus.cc:740] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.885533 1203 consensus_queue.cc:260] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.886075 1203 ts_tablet_manager.cc:1434] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:38.886385 1203 tablet_bootstrap.cc:492] T 2398f176ee044c15841eaa40cb74974c P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.887038 2319 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 47e873c6be06401282f60d87d1a70865
W20251214 20:52:38.887362 2330 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8c4693bad7a485ea769ab075c536b35 consensus state has no leader
I20251214 20:52:38.887570 2330 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41562 (request call id 70) took 1836 ms. Trace:
I20251214 20:52:38.887643 2319 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41562 (request call id 75) took 1724 ms. Trace:
I20251214 20:52:38.888275 1203 tablet_bootstrap.cc:654] T 2398f176ee044c15841eaa40cb74974c P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:38.887678 2330 rpcz_store.cc:276] 1214 20:52:37.051314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:37.058818 (+ 7504us) service_pool.cc:225] Handling call
1214 20:52:37.059409 (+ 591us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:37.059647 (+ 238us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:37.059889 (+ 242us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:37.059978 (+ 89us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:37.306163 (+246185us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:38.887503 (+1581340us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:37.061650 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:37.061774 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:37.061782 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:37.061785 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:37.061910 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:37.061928 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:37.061933 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:37.062003 (+ 70us) tablet.cc:699] Row locks acquired
1214 20:52:37.062006 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:37.062046 (+ 40us) write_op.cc:270] Start()
1214 20:52:37.062087 (+ 41us) write_op.cc:276] Timestamp: P: 1765745557061954 usec, L: 0
1214 20:52:37.062091 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:37.062310 (+ 219us) log.cc:844] Serialized 405 byte log entry
1214 20:52:37.304630 (+242320us) op_driver.cc:464] REPLICATION: finished
1214 20:52:37.304818 (+ 188us) write_op.cc:301] APPLY: starting
1214 20:52:37.304841 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:37.304898 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:37.304932 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:37.305011 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:37.305052 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:37.305060 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:37.305172 (+ 112us) log.cc:844] Serialized 23 byte log entry
1214 20:52:37.305249 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:37.305276 (+ 27us) write_op.cc:454] Released schema lock
1214 20:52:37.305285 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:37.305371 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:37.305375 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":128,"apply.run_cpu_time_us":538,"apply.run_wall_time_us":538,"num_ops":1,"prepare.queue_time_us":1047,"prepare.run_cpu_time_us":783,"prepare.run_wall_time_us":783,"raft.queue_time_us":1864,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":269,"replication_time_us":242495}]]}
I20251214 20:52:38.887725 2319 rpcz_store.cc:276] 1214 20:52:37.162898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:37.547785 (+384887us) service_pool.cc:225] Handling call
1214 20:52:37.594565 (+ 46780us) spinlock_profiling.cc:243] Waited 46.3 ms on lock 0x616000050854. stack: 00007fba1fa45ee6 00007fba1fa45ca0 00007fba1ea213ad 0000000000984f94 0000000000984f5d 00007fba2ed6b6bd 00007fba2ed6b7b3 00007fba2b932521 00007fba335f196d 00007fba3377f1b6 00007fba2a28a486 00007fba2a28a1b5 00007fba26a0456e 00007fba26a0344e 00007fba26a09685 00007fba26a09faf
1214 20:52:37.595016 (+ 451us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:37.595197 (+ 181us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:37.595267 (+ 70us) catalog_manager.cc:2824] Locking tablets
1214 20:52:38.059563 (+464296us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:38.068800 (+ 9237us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:38.068851 (+ 51us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:38.886959 (+818108us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:38.887587 (+ 628us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:38.060397 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:38.060535 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:38.060541 (+ 6us) write_op.cc:435] Acquired schema lock
1214 20:52:38.060543 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:38.060693 (+ 150us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:38.060706 (+ 13us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:38.060711 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:38.060817 (+ 106us) tablet.cc:699] Row locks acquired
1214 20:52:38.060819 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:38.060856 (+ 37us) write_op.cc:270] Start()
1214 20:52:38.060895 (+ 39us) write_op.cc:276] Timestamp: P: 1765745558060774 usec, L: 0
1214 20:52:38.060899 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:38.061084 (+ 185us) log.cc:844] Serialized 725 byte log entry
1214 20:52:38.067304 (+ 6220us) op_driver.cc:464] REPLICATION: finished
1214 20:52:38.068477 (+ 1173us) write_op.cc:301] APPLY: starting
1214 20:52:38.068486 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:38.068514 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:38.068533 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:38.068593 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:38.068618 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:38.068625 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:38.068678 (+ 53us) log.cc:844] Serialized 29 byte log entry
1214 20:52:38.068723 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:38.068738 (+ 15us) write_op.cc:454] Released schema lock
1214 20:52:38.068743 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":106594176,"child_traces":[["op",{"apply.queue_time_us":1141,"apply.run_cpu_time_us":316,"apply.run_wall_time_us":316,"num_ops":2,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":780,"prepare.run_wall_time_us":780,"raft.queue_time_us":56,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":6388,"thread_start_us":12474,"threads_started":1}]]}
I20251214 20:52:38.889189 2320 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 3a23561b9f4e48bbbd796e14fafec086
I20251214 20:52:38.889667 2320 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:41562 (request call id 76) took 1723 ms. Trace:
I20251214 20:52:38.889783 2320 rpcz_store.cc:276] 1214 20:52:37.166441 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:37.763822 (+597381us) service_pool.cc:225] Handling call
1214 20:52:37.764742 (+ 920us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:37.764908 (+ 166us) catalog_manager.cc:2815] Modifying in-memory table state
1214 20:52:37.764999 (+ 91us) catalog_manager.cc:2824] Locking tablets
1214 20:52:38.056353 (+291354us) catalog_manager.cc:2838] Removing table and tablets from system table
1214 20:52:38.068652 (+ 12299us) catalog_manager.cc:2856] Removing table from by-name map
1214 20:52:38.068752 (+ 100us) catalog_manager.cc:2872] Committing in-memory state
1214 20:52:38.889117 (+820365us) catalog_manager.cc:2876] Aborting table tasks
1214 20:52:38.889628 (+ 511us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:38.057428 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:38.057610 (+ 182us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:38.057617 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:38.057620 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:38.057788 (+ 168us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:38.057806 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:38.057812 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:38.057981 (+ 169us) tablet.cc:699] Row locks acquired
1214 20:52:38.057984 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:38.058031 (+ 47us) write_op.cc:270] Start()
1214 20:52:38.058074 (+ 43us) write_op.cc:276] Timestamp: P: 1765745558057947 usec, L: 0
1214 20:52:38.058079 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:38.058344 (+ 265us) log.cc:844] Serialized 790 byte log entry
1214 20:52:38.067231 (+ 8887us) op_driver.cc:464] REPLICATION: finished
1214 20:52:38.067858 (+ 627us) write_op.cc:301] APPLY: starting
1214 20:52:38.067891 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:38.067954 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:38.067988 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:38.068080 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:38.068120 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1214 20:52:38.068127 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:38.068218 (+ 91us) log.cc:844] Serialized 29 byte log entry
1214 20:52:38.068285 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:38.068310 (+ 25us) write_op.cc:454] Released schema lock
1214 20:52:38.068318 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:38.068397 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:38.068400 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":558,"apply.run_cpu_time_us":537,"apply.run_wall_time_us":537,"num_ops":2,"prepare.queue_time_us":65,"prepare.run_cpu_time_us":1037,"prepare.run_wall_time_us":1038,"raft.queue_time_us":363,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":9,"replication_time_us":9117}]]}
W20251214 20:52:38.891350 2326 catalog_manager.cc:5914] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c8af3d2720a499eb51bcfb213bcdbd1 consensus state has no leader
I20251214 20:52:38.891520 2326 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41562 (request call id 64) took 1847 ms. Trace:
I20251214 20:52:38.891615 2326 rpcz_store.cc:276] 1214 20:52:37.044273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:37.044337 (+ 64us) service_pool.cc:225] Handling call
1214 20:52:37.064595 (+ 20258us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:37.108616 (+ 44021us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:37.108802 (+ 186us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:37.108883 (+ 81us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:37.337993 (+229110us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:38.891485 (+1553492us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:37.111119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:37.111232 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:37.111236 (+ 4us) write_op.cc:435] Acquired schema lock
1214 20:52:37.111237 (+ 1us) tablet.cc:660] Decoding operations
1214 20:52:37.111302 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:37.111309 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:37.111312 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:37.111351 (+ 39us) tablet.cc:699] Row locks acquired
1214 20:52:37.111353 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:37.111377 (+ 24us) write_op.cc:270] Start()
1214 20:52:37.111398 (+ 21us) write_op.cc:276] Timestamp: P: 1765745557111282 usec, L: 0
1214 20:52:37.111401 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:37.111530 (+ 129us) log.cc:844] Serialized 405 byte log entry
1214 20:52:37.333726 (+222196us) op_driver.cc:464] REPLICATION: finished
1214 20:52:37.335590 (+ 1864us) write_op.cc:301] APPLY: starting
1214 20:52:37.335598 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:37.335621 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:37.335662 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:37.335692 (+ 30us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:37.335714 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:37.335721 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:37.335789 (+ 68us) log.cc:844] Serialized 23 byte log entry
1214 20:52:37.335823 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:37.335837 (+ 14us) write_op.cc:454] Released schema lock
1214 20:52:37.335842 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:37.335897 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:37.335900 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":896,"child_traces":[["op",{"apply.queue_time_us":1834,"apply.run_cpu_time_us":300,"apply.run_wall_time_us":300,"num_ops":1,"prepare.queue_time_us":1918,"prepare.run_cpu_time_us":495,"prepare.run_wall_time_us":494,"raft.queue_time_us":31,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":222310,"spinlock_wait_cycles":17664}]]}
I20251214 20:52:38.892144 1203 tablet_bootstrap.cc:492] T 2398f176ee044c15841eaa40cb74974c P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:38.892316 1203 ts_tablet_manager.cc:1403] T 2398f176ee044c15841eaa40cb74974c P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20251214 20:52:38.893313 1203 raft_consensus.cc:359] T 2398f176ee044c15841eaa40cb74974c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.893533 1203 raft_consensus.cc:385] T 2398f176ee044c15841eaa40cb74974c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:38.893585 1203 raft_consensus.cc:740] T 2398f176ee044c15841eaa40cb74974c P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:38.893877 1203 consensus_queue.cc:260] T 2398f176ee044c15841eaa40cb74974c P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.894634 1203 ts_tablet_manager.cc:1434] T 2398f176ee044c15841eaa40cb74974c P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251214 20:52:38.894958 1203 tablet_bootstrap.cc:492] T 5e92f54daf4f495fb6f026c179852362 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:38.897024 1019 tablet_service.cc:1552] Processing DeleteTablet for tablet 76d16819e133434eb1c5a65fffb1f03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:38.897457 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e0b89d02c04625b8e31910d60d5d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:38.897500 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a23561b9f4e48bbbd796e14fafec086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:38.897662 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e873c6be06401282f60d87d1a70865 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:38.898360 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 76d16819e133434eb1c5a65fffb1f03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:38.898702 878 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e0b89d02c04625b8e31910d60d5d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:38.898878 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e873c6be06401282f60d87d1a70865 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:38.899313 1203 tablet_bootstrap.cc:654] T 5e92f54daf4f495fb6f026c179852362 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:38.899859 848 consensus_peers.cc:597] T 43fe3bb1ed3941ce9078376f8f478408 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:38.899868 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a23561b9f4e48bbbd796e14fafec086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:38.901623 2316 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:41562 (request call id 65) took 1853 ms. Trace:
I20251214 20:52:38.901769 2316 rpcz_store.cc:276] 1214 20:52:37.048401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:37.048463 (+ 62us) service_pool.cc:225] Handling call
1214 20:52:37.060662 (+ 12199us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1214 20:52:37.060865 (+ 203us) catalog_manager.cc:3708] Apply alter schema
1214 20:52:37.061027 (+ 162us) catalog_manager.cc:3740] Acquired catalog manager lock
1214 20:52:37.061099 (+ 72us) catalog_manager.cc:3898] Updating metadata on disk
1214 20:52:37.306262 (+245163us) catalog_manager.cc:3929] Committing alterations to in-memory state
1214 20:52:38.901541 (+1595279us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:37.062458 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:37.062547 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:37.062552 (+ 5us) write_op.cc:435] Acquired schema lock
1214 20:52:37.062554 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:37.062629 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:37.062637 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:37.062642 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1214 20:52:37.062691 (+ 49us) tablet.cc:699] Row locks acquired
1214 20:52:37.062694 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:37.062719 (+ 25us) write_op.cc:270] Start()
1214 20:52:37.062745 (+ 26us) write_op.cc:276] Timestamp: P: 1765745557062632 usec, L: 0
1214 20:52:37.062748 (+ 3us) op_driver.cc:348] REPLICATION: starting
1214 20:52:37.062895 (+ 147us) log.cc:844] Serialized 405 byte log entry
1214 20:52:37.304717 (+241822us) op_driver.cc:464] REPLICATION: finished
1214 20:52:37.305450 (+ 733us) write_op.cc:301] APPLY: starting
1214 20:52:37.305460 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:37.305486 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:37.305513 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:37.305556 (+ 43us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:37.305581 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:37.305588 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:37.305637 (+ 49us) log.cc:844] Serialized 23 byte log entry
1214 20:52:37.305672 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:37.305683 (+ 11us) write_op.cc:454] Released schema lock
1214 20:52:37.305688 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:37.305744 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:37.305747 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5686,"child_traces":[["op",{"apply.queue_time_us":701,"apply.run_cpu_time_us":288,"apply.run_wall_time_us":288,"num_ops":1,"prepare.queue_time_us":988,"prepare.run_cpu_time_us":523,"prepare.run_wall_time_us":523,"raft.queue_time_us":1106,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":4,"replication_time_us":241948}]]}
I20251214 20:52:38.904328 745 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e0b89d02c04625b8e31910d60d5d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:38.904551 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 47e873c6be06401282f60d87d1a70865 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:38.904698 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 3a23561b9f4e48bbbd796e14fafec086 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:38.909330 3532 tablet_replica.cc:333] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:38.909631 3453 consensus_queue.cc:1048] T 7c2a6c6c0b6e46bcb7658fbeced6f738 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:38.909987 3532 raft_consensus.cc:2243] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:38.911446 3532 raft_consensus.cc:2272] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:38.913460 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 76d16819e133434eb1c5a65fffb1f03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76d16819e133434eb1c5a65fffb1f03f already in progress: creating tablet
I20251214 20:52:38.915081 3453 raft_consensus.cc:493] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.915326 3453 raft_consensus.cc:515] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.916069 3453 leader_election.cc:290] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.916711 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cea9423c40044e4cabb8bbd755a46688" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:38.917164 3532 ts_tablet_manager.cc:1916] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:38.917213 848 leader_election.cc:343] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.923017 1048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cea9423c40044e4cabb8bbd755a46688" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.923343 1048 raft_consensus.cc:2468] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.925837 851 leader_election.cc:304] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.927850 3111 raft_consensus.cc:2804] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.927994 3111 raft_consensus.cc:493] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.928088 3111 raft_consensus.cc:3060] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.930771 3111 raft_consensus.cc:515] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:38.931753 2288 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d71efac30d2f4281a3df37e8306d9a9e (table default.Table_6d6fac7bc4f84ebc961567b02b145610 [id=49756930656347f9a28d186f60b347e3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d71efac30d2f4281a3df37e8306d9a9e has no consensus state
I20251214 20:52:38.932219 3111 leader_election.cc:290] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.933483 3371 raft_consensus.cc:493] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.933701 3371 raft_consensus.cc:515] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.933743 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cea9423c40044e4cabb8bbd755a46688" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:38.934505 3371 leader_election.cc:290] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:38.934809 848 leader_election.cc:343] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.935232 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a43ee234b184275a6ab76b82fe5d283" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:38.935873 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a43ee234b184275a6ab76b82fe5d283" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.936430 991 leader_election.cc:343] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.936887 989 leader_election.cc:343] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.936980 989 leader_election.cc:304] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:38.938294 3371 raft_consensus.cc:2749] T 5a43ee234b184275a6ab76b82fe5d283 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:38.938784 2777 raft_consensus.cc:493] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.938905 2777 raft_consensus.cc:515] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.939493 2777 leader_election.cc:290] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.940873 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a0f669313244e809c2868ca0ccfad65" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:38.941941 848 leader_election.cc:343] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:38.942740 848 consensus_peers.cc:597] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:38.946556 3532 ts_tablet_manager.cc:1929] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251214 20:52:38.947564 2777 consensus_queue.cc:1048] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:38.948814 774 raft_consensus.cc:3060] T e59b80ee919b4e1393fe317ed0e3a3d2 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.956583 1049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a0f669313244e809c2868ca0ccfad65" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:38.963589 1049 raft_consensus.cc:2468] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:38.956632 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cea9423c40044e4cabb8bbd755a46688" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:38.964210 1047 raft_consensus.cc:3060] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.967208 1047 raft_consensus.cc:2468] T cea9423c40044e4cabb8bbd755a46688 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:38.967684 3532 log.cc:1199] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/d5e0b89d02c04625b8e31910d60d5d40
I20251214 20:52:38.968132 3532 ts_tablet_manager.cc:1950] T d5e0b89d02c04625b8e31910d60d5d40 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:38.969045 3451 leader_election.cc:290] T 6d535d8f71b44c81bbba6937e804d900 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313)
I20251214 20:52:38.969398 3532 tablet_replica.cc:333] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:38.969668 3532 raft_consensus.cc:2243] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:38.969897 3532 raft_consensus.cc:2272] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:38.970510 3532 ts_tablet_manager.cc:1916] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:38.971048 851 leader_election.cc:304] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:38.971341 3452 raft_consensus.cc:2804] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:38.971414 3452 raft_consensus.cc:493] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:38.971482 3452 raft_consensus.cc:3060] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.971698 2295 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet d5e0b89d02c04625b8e31910d60d5d40 (table default.Table_b72c21c5b00d430ea93eff8bf8bcb228 [id=4d54f97a2cde4143a185eb4af35bb07d]) successfully deleted
I20251214 20:52:38.974215 3452 raft_consensus.cc:515] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.975159 3452 leader_election.cc:290] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:38.975601 3039 raft_consensus.cc:493] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:38.975737 3039 raft_consensus.cc:515] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:38.976329 3039 leader_election.cc:290] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:38.977008 3532 ts_tablet_manager.cc:1929] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:38.979501 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a0f669313244e809c2868ca0ccfad65" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
W20251214 20:52:38.980008 848 leader_election.cc:343] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:38.986452 3533 tablet_replica.cc:333] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
I20251214 20:52:38.986905 3533 raft_consensus.cc:2243] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:38.990108 3533 raft_consensus.cc:2272] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:38.993510 3533 ts_tablet_manager.cc:1916] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:39.006629 1206 tablet_bootstrap.cc:492] T cc8d725e6ac440899fc1cde2f7b546aa P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:39.006850 1206 ts_tablet_manager.cc:1403] T cc8d725e6ac440899fc1cde2f7b546aa P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.135s user 0.004s sys 0.014s
I20251214 20:52:39.007804 1206 raft_consensus.cc:359] T cc8d725e6ac440899fc1cde2f7b546aa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.008028 1206 raft_consensus.cc:385] T cc8d725e6ac440899fc1cde2f7b546aa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.007968 851 leader_election.cc:304] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:39.008097 1206 raft_consensus.cc:740] T cc8d725e6ac440899fc1cde2f7b546aa P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.008330 1206 consensus_queue.cc:260] T cc8d725e6ac440899fc1cde2f7b546aa P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.008463 3452 raft_consensus.cc:2804] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:39.008577 3452 raft_consensus.cc:697] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:39.008991 3452 consensus_queue.cc:237] T cea9423c40044e4cabb8bbd755a46688 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.012213 2316 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_d918a106076c4b5e91c051527640dc3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:39.012308 2326 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_b5a17650b2fa472d8d040e2f69ceb937"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:39.012588 1206 ts_tablet_manager.cc:1434] T cc8d725e6ac440899fc1cde2f7b546aa P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20251214 20:52:39.013108 2316 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d918a106076c4b5e91c051527640dc3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:39.013273 2326 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5a17650b2fa472d8d040e2f69ceb937 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:39.016734 2319 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_3d34e76a60c641858e292fea246534ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:39.020679 2320 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_91fa15aeeca94237aa90371d170b2318"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:38.980438 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d171301941fa43908e05984c3b016007" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
W20251214 20:52:39.021220 2319 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d34e76a60c641858e292fea246534ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251214 20:52:39.021368 2320 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91fa15aeeca94237aa90371d170b2318 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:39.021576 3532 log.cc:1199] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/3a23561b9f4e48bbbd796e14fafec086
I20251214 20:52:39.022186 3532 ts_tablet_manager.cc:1950] T 3a23561b9f4e48bbbd796e14fafec086 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
W20251214 20:52:39.026262 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 76d16819e133434eb1c5a65fffb1f03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76d16819e133434eb1c5a65fffb1f03f already in progress: creating tablet
W20251214 20:52:39.026571 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet d5e0b89d02c04625b8e31910d60d5d40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5e0b89d02c04625b8e31910d60d5d40 already in progress: creating tablet
I20251214 20:52:39.026633 2295 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 3a23561b9f4e48bbbd796e14fafec086 (table default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3 [id=f1338e81f4ec4dce87130f2ddbe98041]) successfully deleted
W20251214 20:52:39.026821 2291 catalog_manager.cc:4965] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): delete failed for tablet 47e873c6be06401282f60d87d1a70865 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e873c6be06401282f60d87d1a70865 already in progress: creating tablet
W20251214 20:52:39.026854 2295 catalog_manager.cc:4965] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): delete failed for tablet 47e873c6be06401282f60d87d1a70865 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e873c6be06401282f60d87d1a70865 already in progress: creating tablet
I20251214 20:52:39.027021 1047 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a0f669313244e809c2868ca0ccfad65" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:39.027276 1047 raft_consensus.cc:3060] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:39.030229 1047 raft_consensus.cc:2468] T 3a0f669313244e809c2868ca0ccfad65 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
I20251214 20:52:39.031157 1206 tablet_bootstrap.cc:492] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:39.032927 774 raft_consensus.cc:3060] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:38.990474 1203 tablet_bootstrap.cc:492] T 5e92f54daf4f495fb6f026c179852362 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:39.036065 1203 ts_tablet_manager.cc:1403] T 5e92f54daf4f495fb6f026c179852362 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.008s
I20251214 20:52:39.037118 3371 raft_consensus.cc:493] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.037277 3371 raft_consensus.cc:515] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.037935 3371 leader_election.cc:290] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:39.038301 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6302145a6e7b4297a58d9d7f3ad02dce" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:39.038489 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6302145a6e7b4297a58d9d7f3ad02dce" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:39.038463 1203 raft_consensus.cc:359] T 5e92f54daf4f495fb6f026c179852362 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:39.038899 989 leader_election.cc:343] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:39.039208 991 leader_election.cc:343] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.038926 1203 raft_consensus.cc:385] T 5e92f54daf4f495fb6f026c179852362 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.039332 991 leader_election.cc:304] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.039394 1203 raft_consensus.cc:740] T 5e92f54daf4f495fb6f026c179852362 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.039819 3371 raft_consensus.cc:2749] T 6302145a6e7b4297a58d9d7f3ad02dce P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:39.039842 1203 consensus_queue.cc:260] T 5e92f54daf4f495fb6f026c179852362 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.044821 2325 catalog_manager.cc:6199] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: default.Table_b7bd23d0e40346ca98a98ca28dced554 [id=086ee2aba6724a93b4bb736886fe9615] alter complete (version 2)
W20251214 20:52:39.047516 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 76d16819e133434eb1c5a65fffb1f03f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76d16819e133434eb1c5a65fffb1f03f already in progress: creating tablet
I20251214 20:52:39.048481 1206 tablet_bootstrap.cc:654] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:39.048552 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet d5e0b89d02c04625b8e31910d60d5d40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5e0b89d02c04625b8e31910d60d5d40 already in progress: creating tablet
W20251214 20:52:39.049546 2288 catalog_manager.cc:4965] TS 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): delete failed for tablet 47e873c6be06401282f60d87d1a70865 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47e873c6be06401282f60d87d1a70865 already in progress: creating tablet
I20251214 20:52:39.050117 3529 tablet_replica.cc:333] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212: stopping tablet replica
I20251214 20:52:39.050454 3529 raft_consensus.cc:2243] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:39.050781 3529 raft_consensus.cc:2272] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:39.053602 3066 server_base.cc:1047] running on GCE node
I20251214 20:52:39.054629 3371 raft_consensus.cc:493] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.054785 3371 raft_consensus.cc:515] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.056403 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "489a43e7a1da41f4b1a8e80198eec212" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:39.057273 991 leader_election.cc:343] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.057461 3305 raft_consensus.cc:493] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.057574 3305 raft_consensus.cc:515] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:39.058152 3305 leader_election.cc:290] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:39.058638 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ec7824f5eff4cc59c45acd3b3b3d194" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:39.058705 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ec7824f5eff4cc59c45acd3b3b3d194" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:39.059005 3371 leader_election.cc:290] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:39.059486 989 leader_election.cc:343] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:39.059782 991 leader_election.cc:343] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.059911 991 leader_election.cc:304] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.061028 3305 raft_consensus.cc:2749] T 5ec7824f5eff4cc59c45acd3b3b3d194 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:39.061510 3287 raft_consensus.cc:493] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.061527 3486 raft_consensus.cc:493] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2e8fd1cb731446d88f8e510c7a52999)
I20251214 20:52:39.061733 3287 raft_consensus.cc:515] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.061831 3486 raft_consensus.cc:515] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.062316 3287 leader_election.cc:290] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:39.062878 3486 leader_election.cc:290] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:39.063046 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e68bf6b1a2974957adae70449334df64" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
W20251214 20:52:39.063648 848 leader_election.cc:343] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.065673 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d171301941fa43908e05984c3b016007" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:39.067301 712 leader_election.cc:304] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.067896 3486 raft_consensus.cc:2749] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:39.068782 3539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251214 20:52:39.066905 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "489a43e7a1da41f4b1a8e80198eec212" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:39.070272 3066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20251214 20:52:39.073143 713 leader_election.cc:343] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): Not found: Tablet not found: 8d9f74fb2c6244a6887e3c922262bbab
W20251214 20:52:39.072930 989 leader_election.cc:343] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.073722 989 leader_election.cc:304] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.074221 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e68bf6b1a2974957adae70449334df64" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:39.074446 1054 raft_consensus.cc:2468] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:39.074534 3371 raft_consensus.cc:2749] T 489a43e7a1da41f4b1a8e80198eec212 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:39.081521 1202 tablet_bootstrap.cc:492] T 7242be189ca54c9182181e33d6743e6b P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
W20251214 20:52:39.081508 712 leader_election.cc:343] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 8d9f74fb2c6244a6887e3c922262bbab
I20251214 20:52:39.081653 712 leader_election.cc:304] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.081732 1202 ts_tablet_manager.cc:1403] T 7242be189ca54c9182181e33d6743e6b P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.342s user 0.010s sys 0.026s
I20251214 20:52:39.082073 3039 raft_consensus.cc:2749] T 8d9f74fb2c6244a6887e3c922262bbab P 074399864fb2470fa9e2ee87c1fe2212 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:39.082662 1202 raft_consensus.cc:359] T 7242be189ca54c9182181e33d6743e6b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:39.082878 1202 raft_consensus.cc:385] T 7242be189ca54c9182181e33d6743e6b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.082928 1202 raft_consensus.cc:740] T 7242be189ca54c9182181e33d6743e6b P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.083153 1202 consensus_queue.cc:260] T 7242be189ca54c9182181e33d6743e6b P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:39.084245 1202 ts_tablet_manager.cc:1434] T 7242be189ca54c9182181e33d6743e6b P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251214 20:52:39.084686 1202 tablet_bootstrap.cc:492] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:39.086818 1202 tablet_bootstrap.cc:654] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:39.091885 3529 ts_tablet_manager.cc:1916] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:39.093667 851 leader_election.cc:304] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:39.094050 3111 raft_consensus.cc:2804] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:39.094208 3111 raft_consensus.cc:697] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:39.094497 3111 consensus_queue.cc:237] T 3a0f669313244e809c2868ca0ccfad65 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.099591 3533 ts_tablet_manager.cc:1929] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:39.103391 3552 tablet.cc:1722] T 566c6e4761d5486d964e276d151e32b0 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251214 20:52:39.103049 3533 log.cc:1199] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/3a23561b9f4e48bbbd796e14fafec086
I20251214 20:52:39.105027 851 leader_election.cc:304] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:39.105484 1206 tablet_bootstrap.cc:492] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:39.105660 1206 ts_tablet_manager.cc:1403] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.005s
I20251214 20:52:39.105903 3111 raft_consensus.cc:2804] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:39.105986 3111 raft_consensus.cc:493] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:39.106060 3111 raft_consensus.cc:3060] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:39.106585 1206 raft_consensus.cc:359] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:39.106807 1206 raft_consensus.cc:385] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.106909 1206 raft_consensus.cc:740] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.107204 1206 consensus_queue.cc:260] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } }
I20251214 20:52:39.107187 3533 ts_tablet_manager.cc:1950] T 3a23561b9f4e48bbbd796e14fafec086 P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:39.107884 1206 ts_tablet_manager.cc:1434] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:39.108175 1206 tablet_bootstrap.cc:492] T 812642cfcff940f899ef9d3644909ec4 P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:39.109477 1206 tablet_bootstrap.cc:654] T 812642cfcff940f899ef9d3644909ec4 P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:39.111236 1203 ts_tablet_manager.cc:1434] T 5e92f54daf4f495fb6f026c179852362 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.075s user 0.002s sys 0.001s
I20251214 20:52:39.112138 3111 raft_consensus.cc:515] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.114662 3066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251214 20:52:39.120040 1202 tablet_bootstrap.cc:492] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:39.123100 1202 ts_tablet_manager.cc:1403] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.003s
I20251214 20:52:39.124034 1202 raft_consensus.cc:359] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.125576 3066 hybrid_clock.cc:648] HybridClock initialized: now 1765745559125483 us; error 75 us; skew 500 ppm
I20251214 20:52:39.126085 3066 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251214 20:52:39.127749 1202 raft_consensus.cc:385] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.128062 1202 raft_consensus.cc:740] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.128149 2291 catalog_manager.cc:4985] TS a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): tablet 3a23561b9f4e48bbbd796e14fafec086 (table default.Table_78cc09d3ca3d4871a4c1f2214e4b3bb3 [id=f1338e81f4ec4dce87130f2ddbe98041]) successfully deleted
I20251214 20:52:39.128430 1202 consensus_queue.cc:260] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.128965 1202 ts_tablet_manager.cc:1434] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20251214 20:52:39.129244 1202 tablet_bootstrap.cc:492] T 3de3864300a94e53a856c6a93e8d6e80 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:39.129042 1203 tablet_bootstrap.cc:492] T 2593f67c7c6441b694d99286e3ba5883 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:39.131525 3555 tablet.cc:1722] T 5157ef5f1fd541efbbd2887bf4505254 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:39.132754 3371 raft_consensus.cc:493] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.132933 3371 raft_consensus.cc:515] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.133754 3371 leader_election.cc:290] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:39.134604 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51366a6aefd54077877ec5d076394d79" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:39.134999 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e68bf6b1a2974957adae70449334df64" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:39.135322 1054 raft_consensus.cc:3060] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:39.135440 991 leader_election.cc:343] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.137364 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51366a6aefd54077877ec5d076394d79" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:39.138025 1054 raft_consensus.cc:2468] T e68bf6b1a2974957adae70449334df64 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
W20251214 20:52:39.138396 989 leader_election.cc:343] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.138516 989 leader_election.cc:304] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.138744 3371 raft_consensus.cc:2749] T 51366a6aefd54077877ec5d076394d79 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:39.114435 848 consensus_peers.cc:597] T a4643effa71d4d18850913a59ae1c75d P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:39.143795 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e68bf6b1a2974957adae70449334df64" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:39.144793 2316 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41562 (request call id 80) took 1950 ms. Trace:
I20251214 20:52:39.144954 2316 rpcz_store.cc:276] 1214 20:52:37.194695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:38.902530 (+1707835us) service_pool.cc:225] Handling call
1214 20:52:39.012122 (+109592us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x616000050854. stack: 00007fba1fa45ee6 00007fba1fa45ca0 00007fba1ea213ad 0000000000984f94 0000000000984f5d 00007fba2ed6b6bd 00007fba2ed6b7b3 00007fba2b932521 00007fba335f196d 00007fba3377e1c3 00007fba2a2887c6 00007fba2a2884f5 00007fba26a0456e 00007fba26a0344e 00007fba26a09685 00007fba26a09faf
1214 20:52:39.013229 (+ 1107us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:39.013758 (+ 529us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:39.120966 (+107208us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:39.144554 (+ 23588us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:39.144682 (+ 128us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:39.043911 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:39.046245 (+ 2334us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:39.046253 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:39.046256 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:39.046457 (+ 201us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:39.046494 (+ 37us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:39.046499 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:39.046657 (+ 158us) tablet.cc:699] Row locks acquired
1214 20:52:39.046660 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:39.046769 (+ 109us) write_op.cc:270] Start()
1214 20:52:39.046840 (+ 71us) write_op.cc:276] Timestamp: P: 1765745559046621 usec, L: 0
1214 20:52:39.046846 (+ 6us) op_driver.cc:348] REPLICATION: starting
1214 20:52:39.053478 (+ 6632us) log.cc:844] Serialized 433 byte log entry
1214 20:52:39.060392 (+ 6914us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x616000050854. stack: 00007fba1fa45ee6 00007fba1fa45ca0 00007fba1ea213ad 0000000000984f94 0000000000984f5d 000000000098492a 00007fba2b904922 00007fba2cce7875 00007fba2cce6399 00007fba2ccee7c0 00007fba2ccee5cd 0000000000a8970c 00007fba1fa8c20f 00007fba1fa8f52b 00007fba1fa8f3a8 0000000000a8970c
1214 20:52:39.120066 (+ 59674us) op_driver.cc:464] REPLICATION: finished
1214 20:52:39.120181 (+ 115us) write_op.cc:301] APPLY: starting
1214 20:52:39.120217 (+ 36us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:39.120283 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:39.120286 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:39.120492 (+ 206us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:39.120537 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:39.120544 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:39.120661 (+ 117us) log.cc:844] Serialized 29 byte log entry
1214 20:52:39.120762 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:39.120790 (+ 28us) write_op.cc:454] Released schema lock
1214 20:52:39.120801 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":251199616,"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":696,"apply.run_wall_time_us":697,"num_ops":2,"prepare.queue_time_us":29091,"prepare.run_cpu_time_us":1638,"prepare.run_wall_time_us":16611,"raft.queue_time_us":8864,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":7,"replication_time_us":73175,"spinlock_wait_cycles":14425344}]]}
I20251214 20:52:39.145709 2316 master_service.cc:954] ReplaceTablet: received request to replace tablet 532c44785f204f988518c6fd628f76ca from {username='slave'} at 127.0.0.1:41562
I20251214 20:52:39.120234 3111 leader_election.cc:290] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:39.151054 774 raft_consensus.cc:3060] T 3bed2c326906486e99300e05033c9c0f P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:39.159982 1202 tablet_bootstrap.cc:654] T 3de3864300a94e53a856c6a93e8d6e80 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:39.160737 2777 consensus_queue.cc:1048] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251214 20:52:39.161821 848 leader_election.cc:343] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.167289 3553 tablet.cc:1722] T 5157ef5f1fd541efbbd2887bf4505254 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:39.168085 851 leader_election.cc:304] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:39.168730 2777 raft_consensus.cc:2804] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:39.168944 2777 raft_consensus.cc:697] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:39.169253 2777 consensus_queue.cc:237] T e68bf6b1a2974957adae70449334df64 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.169656 3451 consensus_queue.cc:1048] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:39.174050 3529 ts_tablet_manager.cc:1929] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:39.175817 2326 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41562 (request call id 79) took 1981 ms. Trace:
I20251214 20:52:39.176036 2326 rpcz_store.cc:276] 1214 20:52:37.194613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:38.892244 (+1697631us) service_pool.cc:225] Handling call
1214 20:52:39.012227 (+119983us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x616000050854. stack: 00007fba1fa45ee6 00007fba1fa45ca0 00007fba1ea213ad 0000000000984f94 0000000000984f5d 00007fba2ed6b6bd 00007fba2ed6b7b3 00007fba2b932521 00007fba335f196d 00007fba3377e1c3 00007fba2a2887c6 00007fba2a2884f5 00007fba26a0456e 00007fba26a0344e 00007fba26a09685 00007fba26a09faf
1214 20:52:39.013382 (+ 1155us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:39.013872 (+ 490us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:39.052471 (+ 38599us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:39.175577 (+123106us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:39.175733 (+ 156us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:39.015164 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:39.015295 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:39.015303 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:39.015306 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:39.015417 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:39.015432 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:39.015437 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:39.015535 (+ 98us) tablet.cc:699] Row locks acquired
1214 20:52:39.015538 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:39.015577 (+ 39us) write_op.cc:270] Start()
1214 20:52:39.015619 (+ 42us) write_op.cc:276] Timestamp: P: 1765745559015440 usec, L: 0
1214 20:52:39.015623 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:39.015795 (+ 172us) log.cc:844] Serialized 433 byte log entry
1214 20:52:39.051551 (+ 35756us) op_driver.cc:464] REPLICATION: finished
1214 20:52:39.051729 (+ 178us) write_op.cc:301] APPLY: starting
1214 20:52:39.051749 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:39.051815 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:39.051850 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:39.052005 (+ 155us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:39.052046 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:39.052054 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:39.052200 (+ 146us) log.cc:844] Serialized 29 byte log entry
1214 20:52:39.052278 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:39.052299 (+ 21us) write_op.cc:454] Released schema lock
1214 20:52:39.052307 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":275064064,"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":683,"apply.run_wall_time_us":684,"num_ops":2,"prepare.queue_time_us":711,"prepare.run_cpu_time_us":7758,"prepare.run_wall_time_us":28336,"raft.queue_time_us":2066,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":35858,"thread_start_us":27529,"threads_started":1}]]}
I20251214 20:52:39.176779 2326 master_service.cc:954] ReplaceTablet: received request to replace tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 from {username='slave'} at 127.0.0.1:41562
I20251214 20:52:39.177275 2326 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41562 (request call id 82) took 1741 ms. Trace:
I20251214 20:52:39.177369 2326 rpcz_store.cc:276] 1214 20:52:37.436141 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:39.176697 (+1740556us) service_pool.cc:225] Handling call
1214 20:52:39.177217 (+ 520us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 20:52:39.177752 2326 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_93fd3f87a7244f2fa3cdb69da758eff7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251214 20:52:39.178350 2326 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93fd3f87a7244f2fa3cdb69da758eff7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251214 20:52:39.182416 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 7fd4e958e6e74f5ab871d40d2c31e93c (DEFAULT_TABLE table=default.Table_5dcef3898e0a47a5bbabc04825813686 [id=b0c3fad79fb9435d9b212af291f87e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.182958 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fd4e958e6e74f5ab871d40d2c31e93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.183393 1019 tablet_service.cc:1505] Processing CreateTablet for tablet f7ede943c2b447709850840a9dd8b66a (DEFAULT_TABLE table=default.Table_7f0a01f4a19a4b0380b81470a328ac40 [id=215847130602486399b282eb95b70565]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.183719 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7ede943c2b447709850840a9dd8b66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.184396 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 265646e2e48a4b96984e14e659d84aa6 (DEFAULT_TABLE table=default.Table_31aa6fe0cd4e481cb7a2d83d712d69f6 [id=3e9b28dd37894345bb254344b43aa491]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.184836 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 265646e2e48a4b96984e14e659d84aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.186746 2316 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e6713a20a82b434f8a8bc122113e26ad: Sending DeleteTablet for 3 replicas of tablet 532c44785f204f988518c6fd628f76ca
I20251214 20:52:39.187275 2316 catalog_manager.cc:6724] ReplaceTablet: tablet 532c44785f204f988518c6fd628f76ca deleted and replaced by tablet e01ddaf994fb457f80b8e2f4d2e8ac08
I20251214 20:52:39.187404 2316 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41562 (request call id 81) took 1880 ms. Trace:
I20251214 20:52:39.188231 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 532c44785f204f988518c6fd628f76ca with delete_type TABLET_DATA_DELETED (replaced by tablet e01ddaf994fb457f80b8e2f4d2e8ac08) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.187999 2316 rpcz_store.cc:276] 1214 20:52:37.307088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:39.145607 (+1838519us) service_pool.cc:225] Handling call
1214 20:52:39.187370 (+ 41763us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:39.160852 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:39.161007 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:39.161014 (+ 7us) write_op.cc:435] Acquired schema lock
1214 20:52:39.161017 (+ 3us) tablet.cc:660] Decoding operations
1214 20:52:39.161149 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:39.161167 (+ 18us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:39.161173 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:39.161264 (+ 91us) tablet.cc:699] Row locks acquired
1214 20:52:39.161267 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:39.161311 (+ 44us) write_op.cc:270] Start()
1214 20:52:39.161349 (+ 38us) write_op.cc:276] Timestamp: P: 1765745559161145 usec, L: 0
1214 20:52:39.161354 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:39.161565 (+ 211us) log.cc:844] Serialized 622 byte log entry
1214 20:52:39.185076 (+ 23511us) op_driver.cc:464] REPLICATION: finished
1214 20:52:39.185198 (+ 122us) write_op.cc:301] APPLY: starting
1214 20:52:39.185219 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:39.185279 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:39.185305 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:39.185440 (+ 135us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:39.185500 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1214 20:52:39.185508 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:39.185598 (+ 90us) log.cc:844] Serialized 29 byte log entry
1214 20:52:39.185674 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:39.185698 (+ 24us) write_op.cc:454] Released schema lock
1214 20:52:39.185705 (+ 7us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:39.185790 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:39.185793 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":65,"apply.run_cpu_time_us":573,"apply.run_wall_time_us":572,"num_ops":2,"prepare.queue_time_us":467,"prepare.run_cpu_time_us":824,"prepare.run_wall_time_us":824,"raft.queue_time_us":483,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":330,"replication_time_us":23690}]]}
I20251214 20:52:39.188503 3532 tablet_replica.cc:333] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:39.188792 2316 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41562:
table { table_name: "default.Table_0c936131512c4d99b3999d74fabf18d3" } new_table_name: "default.Table_276a40485819408d8c90617d61a97d1c" modify_external_catalogs: true
I20251214 20:52:39.189788 3287 raft_consensus.cc:493] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.193051 2319 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41562 (request call id 77) took 2026 ms. Trace:
I20251214 20:52:39.193466 3529 log.cc:1199] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-0/wal/wals/3a23561b9f4e48bbbd796e14fafec086
I20251214 20:52:39.193288 2319 rpcz_store.cc:276] 1214 20:52:37.194341 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:38.888436 (+1694095us) service_pool.cc:225] Handling call
1214 20:52:39.016641 (+128205us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x616000050854. stack: 00007fba1fa45ee6 00007fba1fa45ca0 00007fba1ea213ad 0000000000984f94 0000000000984f5d 00007fba2ed6b6bd 00007fba2ed6b7b3 00007fba2b932521 00007fba335f196d 00007fba3377e1c3 00007fba2a2887c6 00007fba2a2884f5 00007fba26a0456e 00007fba26a0344e 00007fba26a09685 00007fba26a09faf
1214 20:52:39.144374 (+127733us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:39.144942 (+ 568us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:39.166745 (+ 21803us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:39.192695 (+ 25950us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:39.192825 (+ 130us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:39.145990 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:39.146258 (+ 268us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:39.146266 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:39.146268 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:39.146382 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:39.146397 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:39.146402 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:39.146501 (+ 99us) tablet.cc:699] Row locks acquired
1214 20:52:39.146504 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:39.146550 (+ 46us) write_op.cc:270] Start()
1214 20:52:39.146595 (+ 45us) write_op.cc:276] Timestamp: P: 1765745559146385 usec, L: 0
1214 20:52:39.146599 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:39.146850 (+ 251us) log.cc:844] Serialized 433 byte log entry
1214 20:52:39.154735 (+ 7885us) op_driver.cc:464] REPLICATION: finished
1214 20:52:39.155349 (+ 614us) write_op.cc:301] APPLY: starting
1214 20:52:39.155368 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:39.155421 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:39.155464 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:39.155617 (+ 153us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:39.155660 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:39.155668 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:39.155754 (+ 86us) log.cc:844] Serialized 29 byte log entry
1214 20:52:39.155874 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:39.155899 (+ 25us) write_op.cc:454] Released schema lock
1214 20:52:39.155908 (+ 9us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:39.155989 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:39.155992 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":290082176,"child_traces":[["op",{"apply.queue_time_us":539,"apply.run_cpu_time_us":635,"apply.run_wall_time_us":635,"num_ops":2,"prepare.queue_time_us":401,"prepare.run_cpu_time_us":976,"prepare.run_wall_time_us":977,"raft.queue_time_us":2466,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":8099}]]}
I20251214 20:52:39.193980 3529 ts_tablet_manager.cc:1950] T 3a23561b9f4e48bbbd796e14fafec086 P 074399864fb2470fa9e2ee87c1fe2212: Deleting consensus metadata
I20251214 20:52:39.194803 2319 master_service.cc:954] ReplaceTablet: received request to replace tablet 3b47ca1a03944cf1847d3578856dafa7 from {username='slave'} at 127.0.0.1:41562
I20251214 20:52:39.195240 3066 webserver.cc:492] Webserver started at http://127.0.104.62:44579/ using document root <none> and password file <none>
I20251214 20:52:39.195241 2319 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:41562 (request call id 85) took 1748 ms. Trace:
I20251214 20:52:39.195360 2319 rpcz_store.cc:276] 1214 20:52:37.447052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:39.194701 (+1747649us) service_pool.cc:225] Handling call
1214 20:52:39.195198 (+ 497us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251214 20:52:39.196084 2319 master_service.cc:954] ReplaceTablet: received request to replace tablet de9c830cf9c74cff88cb6f42cbb5f7cd from {username='slave'} at 127.0.0.1:41562
I20251214 20:52:39.196556 3066 fs_manager.cc:362] Metadata directory not provided
I20251214 20:52:39.196776 3066 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251214 20:52:39.197762 3532 raft_consensus.cc:2243] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:39.197973 3532 raft_consensus.cc:2272] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:39.198251 3287 raft_consensus.cc:515] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.198901 3532 ts_tablet_manager.cc:1916] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:39.198974 3287 leader_election.cc:290] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:39.200508 2320 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41562 (request call id 78) took 2005 ms. Trace:
I20251214 20:52:39.200637 2320 rpcz_store.cc:276] 1214 20:52:37.194528 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:38.890448 (+1695920us) service_pool.cc:225] Handling call
1214 20:52:39.020615 (+130167us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x616000050854. stack: 00007fba1fa45ee6 00007fba1fa45ca0 00007fba1ea213ad 0000000000984f94 0000000000984f5d 00007fba2ed6b6bd 00007fba2ed6b7b3 00007fba2b932521 00007fba335f196d 00007fba3377e1c3 00007fba2a2887c6 00007fba2a2884f5 00007fba26a0456e 00007fba26a0344e 00007fba26a09685 00007fba26a09faf
1214 20:52:39.175431 (+154816us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:39.175998 (+ 567us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:39.200212 (+ 24214us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:39.200341 (+ 129us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:39.200454 (+ 113us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:39.177425 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:39.177587 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:39.177595 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:39.177597 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:39.177710 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:39.177725 (+ 15us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:39.177730 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:39.177843 (+ 113us) tablet.cc:699] Row locks acquired
1214 20:52:39.177846 (+ 3us) write_op.cc:260] PREPARE: finished
1214 20:52:39.177889 (+ 43us) write_op.cc:270] Start()
1214 20:52:39.177932 (+ 43us) write_op.cc:276] Timestamp: P: 1765745559177719 usec, L: 0
1214 20:52:39.177937 (+ 5us) op_driver.cc:348] REPLICATION: starting
1214 20:52:39.178158 (+ 221us) log.cc:844] Serialized 433 byte log entry
1214 20:52:39.197664 (+ 19506us) op_driver.cc:464] REPLICATION: finished
1214 20:52:39.197968 (+ 304us) write_op.cc:301] APPLY: starting
1214 20:52:39.197990 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:39.198059 (+ 69us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:39.198087 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:39.198230 (+ 143us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:39.198274 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:39.198281 (+ 7us) write_op.cc:312] APPLY: finished
1214 20:52:39.198385 (+ 104us) log.cc:844] Serialized 29 byte log entry
1214 20:52:39.198462 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:39.198488 (+ 26us) write_op.cc:454] Released schema lock
1214 20:52:39.198496 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1214 20:52:39.198578 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:39.198582 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":298583936,"child_traces":[["op",{"apply.queue_time_us":241,"apply.run_cpu_time_us":600,"apply.run_wall_time_us":601,"num_ops":2,"prepare.queue_time_us":780,"prepare.run_cpu_time_us":817,"prepare.run_wall_time_us":819,"replication_time_us":19694}]]}
I20251214 20:52:39.206432 3532 ts_tablet_manager.cc:1929] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:39.207297 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67104e34df294ef5986171eca79cdc72" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:39.209072 3532 log.cc:1199] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/532c44785f204f988518c6fd628f76ca
I20251214 20:52:39.209556 3532 ts_tablet_manager.cc:1950] T 532c44785f204f988518c6fd628f76ca P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:39.210752 2295 catalog_manager.cc:4985] TS 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313): tablet 532c44785f204f988518c6fd628f76ca (table default.Table_066df6ad9e814765a9866cc1dacd0997 [id=42bd9c72a1384926b366a7a76183cc39]) successfully deleted
W20251214 20:52:39.214215 848 leader_election.cc:343] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.215901 1028 tablet_service.cc:1505] Processing CreateTablet for tablet d968ac07d3d5435496a546bdc22568a5 (DEFAULT_TABLE table=default.Table_219b81378b7d4bccbd72bbb421faad87 [id=01ef97e0306a4a8489d400e0cdcf3355]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.216001 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 9f629f102690489fb77e6d860067e0d5 (DEFAULT_TABLE table=default.Table_01b1f692968d4dbe8d92746b6017a8b3 [id=eefab419a1984e92997ff7429a6dd368]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.216172 1019 tablet_service.cc:1505] Processing CreateTablet for tablet 6ffb115c68e44825a607735c62c9239e (DEFAULT_TABLE table=default.Table_f2b3131db41c42bf8c7ebb469e26ee87 [id=6647250a2bef4f9dbbfff5fb5b7703b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.216336 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d968ac07d3d5435496a546bdc22568a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.219467 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 420c4887800948c088a591eadef5972d (DEFAULT_TABLE table=default.Table_d7da480ef0764d87b50b48556d4160e1 [id=dafe29b66e1845b48f068f6d0ddc7c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.223567 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420c4887800948c088a591eadef5972d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.225111 1203 tablet_bootstrap.cc:654] T 2593f67c7c6441b694d99286e3ba5883 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:39.227550 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f629f102690489fb77e6d860067e0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.231257 1203 tablet_bootstrap.cc:492] T 2593f67c7c6441b694d99286e3ba5883 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:39.231444 1203 ts_tablet_manager.cc:1403] T 2593f67c7c6441b694d99286e3ba5883 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.002s
I20251214 20:52:39.232316 1203 raft_consensus.cc:359] T 2593f67c7c6441b694d99286e3ba5883 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.232506 1203 raft_consensus.cc:385] T 2593f67c7c6441b694d99286e3ba5883 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.232560 1203 raft_consensus.cc:740] T 2593f67c7c6441b694d99286e3ba5883 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.232823 1203 consensus_queue.cc:260] T 2593f67c7c6441b694d99286e3ba5883 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.235375 3371 raft_consensus.cc:493] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.235539 3371 raft_consensus.cc:515] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.236232 3371 leader_election.cc:290] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:39.237003 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ffb115c68e44825a607735c62c9239e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.237231 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b3e0a3bab7a47cb8b26f2094f5a1680" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:39.237905 2326 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41562 (request call id 83) took 1801 ms. Trace:
W20251214 20:52:39.238502 989 leader_election.cc:343] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.238035 2326 rpcz_store.cc:276] 1214 20:52:37.436494 (+ 0us) service_pool.cc:168] Inserting onto call queue
1214 20:52:39.177554 (+1741060us) service_pool.cc:225] Handling call
1214 20:52:39.178457 (+ 903us) catalog_manager.cc:2101] Acquired catalog manager lock
1214 20:52:39.178901 (+ 444us) catalog_manager.cc:2147] Created new table and tablet info
1214 20:52:39.237512 (+ 58611us) catalog_manager.cc:2207] Wrote table and tablets to system table
1214 20:52:39.237653 (+ 141us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
1214 20:52:39.237813 (+ 160us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1214 20:52:39.181873 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1214 20:52:39.181983 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
1214 20:52:39.181991 (+ 8us) write_op.cc:435] Acquired schema lock
1214 20:52:39.181993 (+ 2us) tablet.cc:660] Decoding operations
1214 20:52:39.182089 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1214 20:52:39.182101 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1214 20:52:39.182106 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1214 20:52:39.182180 (+ 74us) tablet.cc:699] Row locks acquired
1214 20:52:39.182182 (+ 2us) write_op.cc:260] PREPARE: finished
1214 20:52:39.182219 (+ 37us) write_op.cc:270] Start()
1214 20:52:39.182257 (+ 38us) write_op.cc:276] Timestamp: P: 1765745559182050 usec, L: 0
1214 20:52:39.182261 (+ 4us) op_driver.cc:348] REPLICATION: starting
1214 20:52:39.182441 (+ 180us) log.cc:844] Serialized 433 byte log entry
1214 20:52:39.197742 (+ 15301us) op_driver.cc:464] REPLICATION: finished
1214 20:52:39.198661 (+ 919us) write_op.cc:301] APPLY: starting
1214 20:52:39.198671 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
1214 20:52:39.198704 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
1214 20:52:39.198727 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1214 20:52:39.198844 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
1214 20:52:39.198873 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
1214 20:52:39.198881 (+ 8us) write_op.cc:312] APPLY: finished
1214 20:52:39.198940 (+ 59us) log.cc:844] Serialized 29 byte log entry
1214 20:52:39.198981 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
1214 20:52:39.198996 (+ 15us) write_op.cc:454] Released schema lock
1214 20:52:39.199002 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":887,"apply.run_cpu_time_us":398,"apply.run_wall_time_us":398,"num_ops":2,"prepare.queue_time_us":2272,"prepare.run_cpu_time_us":643,"prepare.run_wall_time_us":643,"replication_time_us":15463,"threads_started":1}]]}
W20251214 20:52:39.215163 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33313, user_credentials={real_user=slave}} blocked reactor thread for 46541.6us
I20251214 20:52:39.237004 1202 tablet_bootstrap.cc:492] T 3de3864300a94e53a856c6a93e8d6e80 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:39.237778 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b3e0a3bab7a47cb8b26f2094f5a1680" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:39.241542 991 leader_election.cc:343] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.241664 991 leader_election.cc:304] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.242116 3371 raft_consensus.cc:2749] T 0b3e0a3bab7a47cb8b26f2094f5a1680 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:39.242416 1202 ts_tablet_manager.cc:1403] T 3de3864300a94e53a856c6a93e8d6e80 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.113s user 0.006s sys 0.004s
I20251214 20:52:39.245595 1103 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:39.247228 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67104e34df294ef5986171eca79cdc72" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:39.247437 1053 raft_consensus.cc:2468] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 0.
I20251214 20:52:39.255208 1202 raft_consensus.cc:359] T 3de3864300a94e53a856c6a93e8d6e80 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.255648 1202 raft_consensus.cc:385] T 3de3864300a94e53a856c6a93e8d6e80 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.255754 1202 raft_consensus.cc:740] T 3de3864300a94e53a856c6a93e8d6e80 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.256067 1202 consensus_queue.cc:260] T 3de3864300a94e53a856c6a93e8d6e80 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.256698 1202 ts_tablet_manager.cc:1434] T 3de3864300a94e53a856c6a93e8d6e80 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251214 20:52:39.257074 1202 tablet_bootstrap.cc:492] T 99663ff0fe094849acefe03db8d43b50 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:39.258070 1028 tablet_service.cc:1505] Processing CreateTablet for tablet cbf485f4814c4e6d88908d9541bc39a0 (DEFAULT_TABLE table=default.Table_cdf0b37254924fa18a712b2bd472fb46 [id=3322c0d4eb1a409da134030cd0a993fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.258597 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf485f4814c4e6d88908d9541bc39a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.260004 1206 tablet_bootstrap.cc:492] T 812642cfcff940f899ef9d3644909ec4 P 074399864fb2470fa9e2ee87c1fe2212: No bootstrap required, opened a new log
I20251214 20:52:39.260253 1206 ts_tablet_manager.cc:1403] T 812642cfcff940f899ef9d3644909ec4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent bootstrapping tablet: real 0.152s user 0.009s sys 0.001s
I20251214 20:52:39.261075 1206 raft_consensus.cc:359] T 812642cfcff940f899ef9d3644909ec4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.261474 1206 raft_consensus.cc:385] T 812642cfcff940f899ef9d3644909ec4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.261662 1206 raft_consensus.cc:740] T 812642cfcff940f899ef9d3644909ec4 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074399864fb2470fa9e2ee87c1fe2212, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.262161 1206 consensus_queue.cc:260] T 812642cfcff940f899ef9d3644909ec4 P 074399864fb2470fa9e2ee87c1fe2212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.262225 1035 tablet_service.cc:1505] Processing CreateTablet for tablet d71efac30d2f4281a3df37e8306d9a9e (DEFAULT_TABLE table=default.Table_6d6fac7bc4f84ebc961567b02b145610 [id=49756930656347f9a28d186f60b347e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.262666 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d71efac30d2f4281a3df37e8306d9a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.263942 1206 ts_tablet_manager.cc:1434] T 812642cfcff940f899ef9d3644909ec4 P 074399864fb2470fa9e2ee87c1fe2212: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251214 20:52:39.265614 1203 ts_tablet_manager.cc:1434] T 2593f67c7c6441b694d99286e3ba5883 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20251214 20:52:39.266813 1203 tablet_bootstrap.cc:492] T 71b8588773c442fa9add357455fc9d7d P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
W20251214 20:52:39.269523 2288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42865: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20251214 20:52:39.272125 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 03eebe18b4ba449a9dcdc01567b13f9f (DEFAULT_TABLE table=default.Table_7bae2524af7c46b4a8c7df285f89970e [id=3ba9897e15214a7ea69d6c6db3350583]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.272534 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03eebe18b4ba449a9dcdc01567b13f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.274544 1017 tablet_service.cc:1505] Processing CreateTablet for tablet 4706940600084641a836cc819eaf9d9e (DEFAULT_TABLE table=default.Table_1774691d06214e3cbbb1d81b22cbf6c0 [id=55ab757055bb424d8a23945c0655598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.274953 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4706940600084641a836cc819eaf9d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.269866 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 532c44785f204f988518c6fd628f76ca with delete_type TABLET_DATA_DELETED (replaced by tablet e01ddaf994fb457f80b8e2f4d2e8ac08) from {username='slave'} at 127.0.0.1:33546
I20251214 20:52:39.269996 745 tablet_service.cc:1505] Processing CreateTablet for tablet 6ffb115c68e44825a607735c62c9239e (DEFAULT_TABLE table=default.Table_f2b3131db41c42bf8c7ebb469e26ee87 [id=6647250a2bef4f9dbbfff5fb5b7703b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.275718 1203 tablet_bootstrap.cc:654] T 71b8588773c442fa9add357455fc9d7d P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:39.275974 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ffb115c68e44825a607735c62c9239e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.270298 746 tablet_service.cc:1505] Processing CreateTablet for tablet d968ac07d3d5435496a546bdc22568a5 (DEFAULT_TABLE table=default.Table_219b81378b7d4bccbd72bbb421faad87 [id=01ef97e0306a4a8489d400e0cdcf3355]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.270515 742 tablet_service.cc:1505] Processing CreateTablet for tablet d71efac30d2f4281a3df37e8306d9a9e (DEFAULT_TABLE table=default.Table_6d6fac7bc4f84ebc961567b02b145610 [id=49756930656347f9a28d186f60b347e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.280019 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d968ac07d3d5435496a546bdc22568a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.270604 755 tablet_service.cc:1505] Processing CreateTablet for tablet 420c4887800948c088a591eadef5972d (DEFAULT_TABLE table=default.Table_d7da480ef0764d87b50b48556d4160e1 [id=dafe29b66e1845b48f068f6d0ddc7c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.280305 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420c4887800948c088a591eadef5972d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.281329 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 58c28d6833564805973e5b159ff62cf8 (DEFAULT_TABLE table=default.Table_66cfb79ea6504a84884abdbb156f707f [id=3ea66cbd7ec24cd3926f04929f82bae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.281663 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c28d6833564805973e5b159ff62cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.270689 752 tablet_service.cc:1505] Processing CreateTablet for tablet 7fd4e958e6e74f5ab871d40d2c31e93c (DEFAULT_TABLE table=default.Table_5dcef3898e0a47a5bbabc04825813686 [id=b0c3fad79fb9435d9b212af291f87e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.270881 754 tablet_service.cc:1505] Processing CreateTablet for tablet 9f629f102690489fb77e6d860067e0d5 (DEFAULT_TABLE table=default.Table_01b1f692968d4dbe8d92746b6017a8b3 [id=eefab419a1984e92997ff7429a6dd368]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.270942 740 tablet_service.cc:1505] Processing CreateTablet for tablet 265646e2e48a4b96984e14e659d84aa6 (DEFAULT_TABLE table=default.Table_31aa6fe0cd4e481cb7a2d83d712d69f6 [id=3e9b28dd37894345bb254344b43aa491]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.283674 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d71efac30d2f4281a3df37e8306d9a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.270522 756 tablet_service.cc:1505] Processing CreateTablet for tablet f7ede943c2b447709850840a9dd8b66a (DEFAULT_TABLE table=default.Table_7f0a01f4a19a4b0380b81470a328ac40 [id=215847130602486399b282eb95b70565]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.271203 757 tablet_service.cc:1505] Processing CreateTablet for tablet cbf485f4814c4e6d88908d9541bc39a0 (DEFAULT_TABLE table=default.Table_cdf0b37254924fa18a712b2bd472fb46 [id=3322c0d4eb1a409da134030cd0a993fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.275249 3486 raft_consensus.cc:493] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.288524 3486 raft_consensus.cc:515] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.289222 3486 leader_election.cc:290] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:39.289912 887 tablet_service.cc:1505] Processing CreateTablet for tablet f7ede943c2b447709850840a9dd8b66a (DEFAULT_TABLE table=default.Table_7f0a01f4a19a4b0380b81470a328ac40 [id=215847130602486399b282eb95b70565]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.290234 885 tablet_service.cc:1505] Processing CreateTablet for tablet 9f629f102690489fb77e6d860067e0d5 (DEFAULT_TABLE table=default.Table_01b1f692968d4dbe8d92746b6017a8b3 [id=eefab419a1984e92997ff7429a6dd368]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.290431 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7ede943c2b447709850840a9dd8b66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.290647 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f629f102690489fb77e6d860067e0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.292228 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e185d107619414a8bcc15ad496547dc" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75" is_pre_election: true
I20251214 20:52:39.295460 881 tablet_service.cc:1505] Processing CreateTablet for tablet 265646e2e48a4b96984e14e659d84aa6 (DEFAULT_TABLE table=default.Table_31aa6fe0cd4e481cb7a2d83d712d69f6 [id=3e9b28dd37894345bb254344b43aa491]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.295585 879 tablet_service.cc:1505] Processing CreateTablet for tablet cbf485f4814c4e6d88908d9541bc39a0 (DEFAULT_TABLE table=default.Table_cdf0b37254924fa18a712b2bd472fb46 [id=3322c0d4eb1a409da134030cd0a993fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.295926 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 265646e2e48a4b96984e14e659d84aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.296136 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf485f4814c4e6d88908d9541bc39a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.296478 878 tablet_service.cc:1505] Processing CreateTablet for tablet 6ffb115c68e44825a607735c62c9239e (DEFAULT_TABLE table=default.Table_f2b3131db41c42bf8c7ebb469e26ee87 [id=6647250a2bef4f9dbbfff5fb5b7703b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.296849 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ffb115c68e44825a607735c62c9239e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:39.296967 848 consensus_peers.cc:597] T 1def5f0de5ce4eb0bed36f626c4f0683 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:39.297555 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 016581519f4c4e0d832b49346d55b41c (table default.Table_531eb871a88242c5b91423a2c6abc3b6 [id=9afdb569356648bb84089b1738dfc359], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 016581519f4c4e0d832b49346d55b41c consensus state has no leader
W20251214 20:52:39.297724 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 96a2b444608e42d687eeba2dc0ebb757 (table default.Table_90e94bab31e54a9cb8b567622aa4e37d [id=7c3d3542270e449ca954d068bece3491], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 96a2b444608e42d687eeba2dc0ebb757 consensus state has no leader
I20251214 20:52:39.299252 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 532c44785f204f988518c6fd628f76ca with delete_type TABLET_DATA_DELETED (replaced by tablet e01ddaf994fb457f80b8e2f4d2e8ac08) from {username='slave'} at 127.0.0.1:56932
I20251214 20:52:39.299645 892 tablet_service.cc:1505] Processing CreateTablet for tablet d968ac07d3d5435496a546bdc22568a5 (DEFAULT_TABLE table=default.Table_219b81378b7d4bccbd72bbb421faad87 [id=01ef97e0306a4a8489d400e0cdcf3355]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.299988 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d968ac07d3d5435496a546bdc22568a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.303402 876 tablet_service.cc:1505] Processing CreateTablet for tablet 420c4887800948c088a591eadef5972d (DEFAULT_TABLE table=default.Table_d7da480ef0764d87b50b48556d4160e1 [id=dafe29b66e1845b48f068f6d0ddc7c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.303754 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 420c4887800948c088a591eadef5972d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:39.307662 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet a5d28f09ad174936b47debace0599600 (table default.Table_93730665b1ab4686a0d3e54740fbddc8 [id=0b5581be22d345b3a0fd22c3dce2f207], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a5d28f09ad174936b47debace0599600 consensus state has no leader
I20251214 20:52:39.308173 893 tablet_service.cc:1505] Processing CreateTablet for tablet 7fd4e958e6e74f5ab871d40d2c31e93c (DEFAULT_TABLE table=default.Table_5dcef3898e0a47a5bbabc04825813686 [id=b0c3fad79fb9435d9b212af291f87e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.308519 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fd4e958e6e74f5ab871d40d2c31e93c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:39.309357 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet f6d51d07f7e9450a844be9afddb60d03 (table default.Table_8a02e24feeab4dc6a69f6ca246440bad [id=22d141506a1d401ca05343516b778cd3], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f6d51d07f7e9450a844be9afddb60d03 consensus state has no leader
W20251214 20:52:39.309516 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet a4643effa71d4d18850913a59ae1c75d (table default.Table_679d53b1472341ef8612ca53ac1ebda2 [id=91f3b4dd176348a5aa38c6bdc417918f], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet a4643effa71d4d18850913a59ae1c75d consensus state has no leader
I20251214 20:52:39.310338 1203 tablet_bootstrap.cc:492] T 71b8588773c442fa9add357455fc9d7d P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:39.310472 1203 ts_tablet_manager.cc:1403] T 71b8588773c442fa9add357455fc9d7d P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.002s
W20251214 20:52:39.310560 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 202b1d53cab644c98838b0ee81f29f22 (table default.Table_cddd59a230e04595bffdf9e0a0ed1471 [id=6094596f6c064c5bb1e20aa2b8767b9f], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 202b1d53cab644c98838b0ee81f29f22 consensus state has no leader
W20251214 20:52:39.310815 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 9bd89709abbc49809a2cdb6d0c29fdb8 (table default.Table_c3aa8e3c31cb41dc8281962e7b5c46e1 [id=687eb544b4114bad8d2deb76388e2022], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9bd89709abbc49809a2cdb6d0c29fdb8 consensus state has no leader
I20251214 20:52:39.311239 1203 raft_consensus.cc:359] T 71b8588773c442fa9add357455fc9d7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.311391 1203 raft_consensus.cc:385] T 71b8588773c442fa9add357455fc9d7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.311434 1203 raft_consensus.cc:740] T 71b8588773c442fa9add357455fc9d7d P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.311648 1203 consensus_queue.cc:260] T 71b8588773c442fa9add357455fc9d7d P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.312122 1203 ts_tablet_manager.cc:1434] T 71b8588773c442fa9add357455fc9d7d P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251214 20:52:39.312389 1203 tablet_bootstrap.cc:492] T c3453fc996fb4d2b8cf0f80152098c43 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:39.313566 886 tablet_service.cc:1505] Processing CreateTablet for tablet 4706940600084641a836cc819eaf9d9e (DEFAULT_TABLE table=default.Table_1774691d06214e3cbbb1d81b22cbf6c0 [id=55ab757055bb424d8a23945c0655598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.313956 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4706940600084641a836cc819eaf9d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.314643 1203 tablet_bootstrap.cc:654] T c3453fc996fb4d2b8cf0f80152098c43 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:39.315714 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e185d107619414a8bcc15ad496547dc" candidate_uuid: "074399864fb2470fa9e2ee87c1fe2212" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
I20251214 20:52:39.315778 889 tablet_service.cc:1505] Processing CreateTablet for tablet b0f899161e31463b99847dc46d5cc4ff (DEFAULT_TABLE table=default.Table_36e470f74bfc43368ea54b7bba3d0332 [id=732f7739bc0b4baa9314e669aa55a550]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.316151 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0f899161e31463b99847dc46d5cc4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.317962 885 tablet_service.cc:1505] Processing CreateTablet for tablet d71efac30d2f4281a3df37e8306d9a9e (DEFAULT_TABLE table=default.Table_6d6fac7bc4f84ebc961567b02b145610 [id=49756930656347f9a28d186f60b347e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.318392 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d71efac30d2f4281a3df37e8306d9a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.319455 894 tablet_service.cc:1505] Processing CreateTablet for tablet 58c28d6833564805973e5b159ff62cf8 (DEFAULT_TABLE table=default.Table_66cfb79ea6504a84884abdbb156f707f [id=3ea66cbd7ec24cd3926f04929f82bae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.319789 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c28d6833564805973e5b159ff62cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.320935 879 tablet_service.cc:1505] Processing CreateTablet for tablet 03eebe18b4ba449a9dcdc01567b13f9f (DEFAULT_TABLE table=default.Table_7bae2524af7c46b4a8c7df285f89970e [id=3ba9897e15214a7ea69d6c6db3350583]), partition=RANGE (key) PARTITION UNBOUNDED
I20251214 20:52:39.321327 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03eebe18b4ba449a9dcdc01567b13f9f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251214 20:52:39.321735 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 67104e34df294ef5986171eca79cdc72 (table default.Table_27222912e9214b879d8a30975aba5688 [id=c0729ac770394d7b87ef36e53f85c151], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 67104e34df294ef5986171eca79cdc72 consensus state has no leader
W20251214 20:52:39.322491 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 526f28f9aaad445081d9d69cc20ed7b3 (table default.Table_b87980e2da6a4052b109938fcf0527d8 [id=88d5807cc76646e3a2962b5d36413894], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 526f28f9aaad445081d9d69cc20ed7b3 consensus state has no leader
W20251214 20:52:39.323055 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 0d0f7bd831b4474a90288ac7fa600436 (table default.Table_d8b86c6bc911454fac28b6384cf9ff3c [id=c97bcc9ac37c4413bae618ee1019e069], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0d0f7bd831b4474a90288ac7fa600436 consensus state has no leader
W20251214 20:52:39.323357 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet b87b0871c2454971a15dd5643c155694 (table default.Table_4a9b23e6d6a048d28ad16576ebf33c24 [id=4efcd09629ed4717b4031e916f8b5749], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b87b0871c2454971a15dd5643c155694 consensus state has no leader
I20251214 20:52:39.324640 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 6eca34627bbc4a3f9915be72f3128370 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d535d8f71b44c81bbba6937e804d900) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.324858 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 292a28b12df246a8bc855d81c530746e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:19 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.325007 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet d171301941fa43908e05984c3b016007 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.325125 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 57a6232564194cbba12182f3a29c2ebb with delete_type TABLET_DATA_DELETED (replaced by tablet aaf060ab2f71429a85dda19669e69677) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.325273 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet c3453fc996fb4d2b8cf0f80152098c43 with delete_type TABLET_DATA_DELETED (replaced by tablet ba322cb6ee084bc39788ddd439f5f15d) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.325388 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet dfe50a6219134431b0a025317932609b with delete_type TABLET_DATA_DELETED (replaced by tablet 6b482043af42485eb4c73d6eb864a963) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.325502 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 6b389a63a58d438db41a343db6e04a88 with delete_type TABLET_DATA_DELETED (replaced by tablet f6d51d07f7e9450a844be9afddb60d03) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.325630 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet d2a57409d4f54f17be792e6f3f17d7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.325728 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f97e5bad46343098418e8e34334b892 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.325868 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c61824bacd406280ba03357f9f5b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.325974 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 99663ff0fe094849acefe03db8d43b50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.326051 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 1eff8a23d48c43e2bdbfb34f4bab6edf with delete_type TABLET_DATA_DELETED (replaced by tablet ed9790fbebda4f848291930bf1dd8174) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.326311 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 248e3666fd13434e97378eb5a98dfadd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.326488 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f75657d7bc44f6393c55ab119eb7dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.326630 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 79e621f29c01448399d22615284fccd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 202b1d53cab644c98838b0ee81f29f22) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.326879 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet bd1161eebe884fe6b347bf517269c047 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.327051 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet d5e0b89d02c04625b8e31910d60d5d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.327266 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet c9488c3505f5443998531cf749722be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:25 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.327333 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 880a78b924574181bdd804d599f4f579 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.327472 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e185d107619414a8bcc15ad496547dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.327594 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 727e7b61ae3942098712720d344c5310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.327682 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f9def8e4fd449599820a63845397750 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.327895 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet b2bea617954c475bac7e9b8719faa028 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.327991 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a43ee234b184275a6ab76b82fe5d283 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.328033 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet c7248a2e8355425fa051a197160dbdbb with delete_type TABLET_DATA_DELETED (replaced by tablet e09e0481ebc64479ab0099093079ffe9) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.328171 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f358df4d9414dcdb0d197f434e28118 with delete_type TABLET_DATA_DELETED (replaced by tablet a4bcac304fd14e20b0607c23d1569a31) from {username='slave'} at 127.0.0.1:49254
W20251214 20:52:39.328723 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 8a818ca230564b93ac165d43ff22deb1 (table default.Table_b289a25bf8174d9199c90212ce70271d [id=0a8ffa32e9ce4e3a8f9fa2afa4de69bb], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 8a818ca230564b93ac165d43ff22deb1 consensus state has no leader
I20251214 20:52:39.328992 3532 tablet_replica.cc:333] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:39.329195 3532 raft_consensus.cc:2243] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:39.329357 3532 raft_consensus.cc:2272] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251214 20:52:39.329587 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 438b8ec5e9834442994966c8d83c1a31 (table default.Table_5cffb2d16a4847caa5cf6a2c4c238da8 [id=a8b3bbe26fc14b4bbc8a78c3d1f9df6d], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 438b8ec5e9834442994966c8d83c1a31 consensus state has no leader
W20251214 20:52:39.329945 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet f8e98027ac3f4f36aded3aa5b47e6cad (table default.Table_5c7e3a65102c45a1a23d29b95427fae5 [id=db0d9e33bf054daabddadd1248243609], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f8e98027ac3f4f36aded3aa5b47e6cad consensus state has no leader
W20251214 20:52:39.330256 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet a12e8053c65343758d518ee840dbaee8 (table default.Table_cb922bf98617480aa9363c3f08d918f3 [id=e815640c7ece4361800a22bc749915e9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a12e8053c65343758d518ee840dbaee8 consensus state has no leader
W20251214 20:52:39.330503 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet e0d0e8f35bf44c9a8797b7507dcce3b9 (table default.Table_4649d8b3e43c42a492723fadd4217e4e [id=75cebf8d0aba4aa48c1be51b0abd6b38], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet e0d0e8f35bf44c9a8797b7507dcce3b9 consensus state has no leader
W20251214 20:52:39.330660 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet e1dbaa39efa04086a22cc3908ed691d4 (table default.Table_8f897cb4591a4d5fadbf2b1b7ee7408d [id=4390cb960d564bb5bc7e0cefa5bf3004], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e1dbaa39efa04086a22cc3908ed691d4 consensus state has no leader
W20251214 20:52:39.330947 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet d39e245bc5a04c1186d0973645aa1b54 (table default.Table_abc73c8e0e7e458abdfb7be16dd324e3 [id=0453c668e17b4c1ba55df9ff8fb1fce7], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet d39e245bc5a04c1186d0973645aa1b54 consensus state has no leader
W20251214 20:52:39.331213 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet f891db00595b49ca9e6ae64721e4d744 (table default.Table_c404f3a3b71f4899a233f864e78a7d13 [id=203681807a4b40b0a1f69387cfecff38], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f891db00595b49ca9e6ae64721e4d744 consensus state has no leader
W20251214 20:52:39.331486 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet bea3e9d7334345fa88ededf1be17a434 (table default.Table_27579b7155b140b3bdda5014c1eb57a8 [id=e24a6c9e5c9049edba5e42e22d22a831], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet bea3e9d7334345fa88ededf1be17a434 consensus state has no leader
I20251214 20:52:39.331633 3532 ts_tablet_manager.cc:1916] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251214 20:52:39.331780 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 96157aae81624a4ba9fcf8c3f142911f (table default.Table_adb418bf04534b868c044988fb97e906 [id=d7e403a944e94d0d8c9d6ba115723ea4], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 96157aae81624a4ba9fcf8c3f142911f consensus state has no leader
W20251214 20:52:39.332418 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 2398f176ee044c15841eaa40cb74974c (table default.Table_25630f29e4904aaf86f8f65078bcf81e [id=b1ee89b437fb42a4b5e2c3d386ee0284], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 2398f176ee044c15841eaa40cb74974c consensus state has no leader
I20251214 20:52:39.332465 3533 tablet_replica.cc:333] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999: stopping tablet replica
W20251214 20:52:39.332576 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 6302145a6e7b4297a58d9d7f3ad02dce (table default.Table_1be39fb576af4b2b9e2606b495619863 [id=54e0182cd3b9461aa593a11aa7709002], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6302145a6e7b4297a58d9d7f3ad02dce consensus state has no leader
W20251214 20:52:39.332724 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 5ec7824f5eff4cc59c45acd3b3b3d194 (table default.Table_6cfa43d70d2f426fb58f8ef32476c3ac [id=be1be47d388347e3a6b7a80105688be3], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 5ec7824f5eff4cc59c45acd3b3b3d194 consensus state has no leader
I20251214 20:52:39.334173 1203 tablet_bootstrap.cc:492] T c3453fc996fb4d2b8cf0f80152098c43 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:39.334313 1203 ts_tablet_manager.cc:1403] T c3453fc996fb4d2b8cf0f80152098c43 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20251214 20:52:39.335050 1203 raft_consensus.cc:359] T c3453fc996fb4d2b8cf0f80152098c43 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.335778 1203 raft_consensus.cc:385] T c3453fc996fb4d2b8cf0f80152098c43 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.335851 1203 raft_consensus.cc:740] T c3453fc996fb4d2b8cf0f80152098c43 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.335899 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet f7f06eded6c340c58101fcc644ebcf50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.336058 1203 consensus_queue.cc:260] T c3453fc996fb4d2b8cf0f80152098c43 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.336131 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 1d7aa508b11944aa8e268741c1998a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.336293 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet cab1bed6016243bda0d9bd7f512884c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.336434 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 06ebe9f717bf43108748605cf4f961b7 with delete_type TABLET_DATA_DELETED (replaced by tablet aadba4faca0c465f9eea004f93ee1190) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.336540 1203 ts_tablet_manager.cc:1434] T c3453fc996fb4d2b8cf0f80152098c43 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251214 20:52:39.336589 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 76d16819e133434eb1c5a65fffb1f03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:37 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.336719 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet f98be57ce1b14f20a3afcb79979b6c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.336798 1203 tablet_bootstrap.cc:492] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:39.336851 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 52f5af48440b4d569939c163f365f6fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.336989 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet e7b247b761c94c60ad3a468db371b30b with delete_type TABLET_DATA_DELETED (replaced by tablet 2f75657d7bc44f6393c55ab119eb7dc2) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.337056 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet cbefd8f111084785bafd7e74ac443fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.337265 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 485ae1bdc1f344e08aadb4d00dc0a1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:36 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.337265 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d535d8f71b44c81bbba6937e804d900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.337489 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet a6df388bdfbc40c183c4b14af2336a6a with delete_type TABLET_DATA_DELETED (replaced by tablet 74d2f8e5481d40b39cc4df34b0d4f819) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.337617 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet f342dafae1d04be3ba38f7c0e4f5d058 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:22 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.337617 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet c249c096a41f4b4093eaf9afc6361249 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.337702 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet b8727f302d35471eb2eac1e77b8d966b with delete_type TABLET_DATA_DELETED (replaced by tablet 9f629f102690489fb77e6d860067e0d5) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.337774 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet fa141b2570514c6aacb2850e4bfd1869 with delete_type TABLET_DATA_DELETED (replaced by tablet 02fc0fc09e8641f1bb05c312422e5971) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.337924 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 18e95ff0e0da4c7680f594ba31cdc3a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.338126 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet ed9790fbebda4f848291930bf1dd8174 with delete_type TABLET_DATA_DELETED (replaced by tablet 76d16819e133434eb1c5a65fffb1f03f) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.338131 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b7dd0ab78534a34ba342ec6e4fae129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.339040 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet c8e33ba1149a44c294885d76f676160a with delete_type TABLET_DATA_DELETED (replaced by tablet 1eff8a23d48c43e2bdbfb34f4bab6edf) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.340763 1202 tablet_bootstrap.cc:654] T 99663ff0fe094849acefe03db8d43b50 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:39.340991 1203 tablet_bootstrap.cc:654] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:39.344369 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 2807af0754914688bd7a090b1bc32eb1 (table default.Table_c6c0496d455a4db68c05c0ff8b06493d [id=9d1473df58ee486aa6a55893aca9d54c], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 2807af0754914688bd7a090b1bc32eb1 consensus state has no leader
W20251214 20:52:39.344573 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 4b5360e7ece94fd2a5a6512db6c6c12f (table default.Table_2ca54bf356e24a1ebca6f4a155b83044 [id=7f407f5abea945498cf939a23b0f9319], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 4b5360e7ece94fd2a5a6512db6c6c12f consensus state has no leader
W20251214 20:52:39.346035 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 639cf3b7654a4ae896c6edd8b2ff1844 (table default.Table_15c1781afe3449ef87ef3c655ccca0df [id=02c00037bc944c00b0124d72af62e19b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 639cf3b7654a4ae896c6edd8b2ff1844 consensus state has no leader
I20251214 20:52:39.346802 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 8e11723c042a4637839f1b13bed0935a with delete_type TABLET_DATA_DELETED (replaced by tablet 6e005feeb95b48b9ad33612ab81d5bce) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.346863 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 825baba561ee4a8ebd44f75d8c51706a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:49254
W20251214 20:52:39.347512 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 3a0f669313244e809c2868ca0ccfad65 (table default.Table_d59e2b0d82384da68c1901ddaa419d00 [id=0c56f14b40ac442994e232aa348d64a2], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3a0f669313244e809c2868ca0ccfad65 consensus state has no leader
W20251214 20:52:39.348630 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 894e3f27b7904db994711aa80a9b1600 (table default.Table_4a58233140a046368357556e00275995 [id=f938ded464bd475983e76b9ec24562cc], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 894e3f27b7904db994711aa80a9b1600 consensus state has no leader
I20251214 20:52:39.350659 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet 0a2fbec0543445b7856ef22c934c6840 with delete_type TABLET_DATA_DELETED (replaced by tablet 700ed62ef2e84719a7a8e9879feaf45c) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.350838 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c62ffb71fff43d4812eb86f40ce1c58 with delete_type TABLET_DATA_DELETED (replaced by tablet cbf485f4814c4e6d88908d9541bc39a0) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.351290 1203 tablet_bootstrap.cc:492] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75: No bootstrap required, opened a new log
I20251214 20:52:39.351426 1203 ts_tablet_manager.cc:1403] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
W20251214 20:52:39.351995 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 0dbfe8ee68bf4a2c88489b1f81b36991 (table default.Table_53bf327c948f4cebbe6096266ac301f8 [id=6a279e57008549c593df838211cc9cc1], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0dbfe8ee68bf4a2c88489b1f81b36991 consensus state has no leader
W20251214 20:52:39.352639 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 6e005feeb95b48b9ad33612ab81d5bce (table default.Table_ec8c8f9185be4470a6580c4e141e2227 [id=8872bfac95c64cbba1ccb84d1d9f2483], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 6e005feeb95b48b9ad33612ab81d5bce consensus state has no leader
I20251214 20:52:39.352797 1203 raft_consensus.cc:359] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.352980 1203 raft_consensus.cc:385] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.353031 1203 raft_consensus.cc:740] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63848a9fb78a4013a6f1657621272d75, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.353974 1203 consensus_queue.cc:260] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.355232 3555 tablet.cc:1722] T 658e4fe0d188483b9ceb9eb3b3f20835 P 63848a9fb78a4013a6f1657621272d75: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:39.355705 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet d745583012b4465090d85c75470e0bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:49254
W20251214 20:52:39.356022 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet ba322cb6ee084bc39788ddd439f5f15d (table default.Table_196d0f488e0b42aabd518511a4289dc3 [id=2f57c28c9c624a6ca1986c8831b85169], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ba322cb6ee084bc39788ddd439f5f15d consensus state has no leader
I20251214 20:52:39.356866 712 leader_election.cc:304] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 074399864fb2470fa9e2ee87c1fe2212; no voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.357511 3272 raft_consensus.cc:2749] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251214 20:52:39.357617 2323 catalog_manager.cc:5846] Failed to send AlterTable RPC for tablet 59ccb6c3b35044dc913cce74b5802e38 (table default.Table_d61784eed0b84f8caa485dbe70cdc366 [id=b5622661ea9f40f7ac11f44ae92a5194], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 59ccb6c3b35044dc913cce74b5802e38 consensus state has no leader
I20251214 20:52:39.354981 3533 raft_consensus.cc:2243] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Raft consensus shutting down.
I20251214 20:52:39.358168 3533 raft_consensus.cc:2272] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:39.359079 3533 ts_tablet_manager.cc:1916] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:39.360090 1021 tablet_service.cc:1552] Processing DeleteTablet for tablet 54e5fff1fc84419ab3d3131cb3a12d92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.360194 1022 tablet_service.cc:1552] Processing DeleteTablet for tablet cea9423c40044e4cabb8bbd755a46688 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.360411 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b3e0a3bab7a47cb8b26f2094f5a1680 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:29 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.360939 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 43fe3bb1ed3941ce9078376f8f478408 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.361724 3553 tablet.cc:1722] T 658e4fe0d188483b9ceb9eb3b3f20835 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:39.363014 1203 ts_tablet_manager.cc:1434] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251214 20:52:39.363257 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet a88207c8cc5f4a6886de36c82c6b4c2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.363446 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 1def5f0de5ce4eb0bed36f626c4f0683 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.364167 1203 tablet_bootstrap.cc:492] T 4f358df4d9414dcdb0d197f434e28118 P 63848a9fb78a4013a6f1657621272d75: Bootstrap starting.
I20251214 20:52:39.365640 3533 ts_tablet_manager.cc:1929] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:39.365737 3533 log.cc:1199] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-1/wal/wals/532c44785f204f988518c6fd628f76ca
I20251214 20:52:39.365900 1203 tablet_bootstrap.cc:654] T 4f358df4d9414dcdb0d197f434e28118 P 63848a9fb78a4013a6f1657621272d75: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:39.366183 3533 ts_tablet_manager.cc:1950] T 532c44785f204f988518c6fd628f76ca P a2e8fd1cb731446d88f8e510c7a52999: Deleting consensus metadata
I20251214 20:52:39.367264 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet e68bf6b1a2974957adae70449334df64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.367529 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet c07966670d294a0495ec54d48c7f09cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.368794 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 348d8dcef97042279bc0d5ff31d3c137 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:30 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.368952 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 52c05204502a45e18c2b862e21c6f361 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:27 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.369007 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 3bed2c326906486e99300e05033c9c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:23 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.369138 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 8eb1006b0ba440dcba2f9de04dca2702 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:21 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.369342 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 5b68367f902f4be386788a8d138b5762 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:31 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.369138 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 3915abc53b084854912a8e4119eefc7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:28 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.369151 1017 tablet_service.cc:1552] Processing DeleteTablet for tablet e59b80ee919b4e1393fe317ed0e3a3d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:26 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.369597 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 34df3a34b15b4fe19bd12f785b07a033 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ec7824f5eff4cc59c45acd3b3b3d194) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.371078 2320 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41562:
name: "default.Table_2e126a1f9ccc4668911a1fc3ae8ad4d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251214 20:52:39.372399 774 raft_consensus.cc:3060] T 1e185d107619414a8bcc15ad496547dc P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:39.379707 851 leader_election.cc:304] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:39.380028 3287 raft_consensus.cc:2804] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251214 20:52:39.380160 3287 raft_consensus.cc:493] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251214 20:52:39.380219 3287 raft_consensus.cc:3060] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:39.380805 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c5ee4114cb34540af2c5b332ebe82ab with delete_type TABLET_DATA_DELETED (replaced by tablet 7e0847d58c0d4c5cbd1f9b91fd103763) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.384138 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 054cfd587d19409fbdf7f4848bcaf3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-14 20:52:24 UTC) from {username='slave'} at 127.0.0.1:49254
I20251214 20:52:39.384686 1202 tablet_bootstrap.cc:492] T 99663ff0fe094849acefe03db8d43b50 P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:39.384809 1202 ts_tablet_manager.cc:1403] T 99663ff0fe094849acefe03db8d43b50 P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.128s user 0.003s sys 0.001s
I20251214 20:52:39.384935 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf485f4814c4e6d88908d9541bc39a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.385064 3532 ts_tablet_manager.cc:1929] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251214 20:52:39.385161 3532 log.cc:1199] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75: Deleting WAL directory at /tmp/dist-test-taskM_U6oP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1765745524223978-416-0/minicluster-data/ts-2/wal/wals/6eca34627bbc4a3f9915be72f3128370
I20251214 20:52:39.385488 3532 ts_tablet_manager.cc:1950] T 6eca34627bbc4a3f9915be72f3128370 P 63848a9fb78a4013a6f1657621272d75: Deleting consensus metadata
I20251214 20:52:39.385939 3532 tablet_replica.cc:333] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75: stopping tablet replica
I20251214 20:52:39.386173 3532 raft_consensus.cc:2243] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251214 20:52:39.386345 3532 raft_consensus.cc:2272] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251214 20:52:39.387952 769 raft_consensus.cc:3060] T 5157ef5f1fd541efbbd2887bf4505254 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:39.388700 3451 consensus_queue.cc:1048] T 54e5fff1fc84419ab3d3131cb3a12d92 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:39.389891 1202 raft_consensus.cc:359] T 99663ff0fe094849acefe03db8d43b50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.390048 1202 raft_consensus.cc:385] T 99663ff0fe094849acefe03db8d43b50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.390091 1202 raft_consensus.cc:740] T 99663ff0fe094849acefe03db8d43b50 P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.390314 1202 consensus_queue.cc:260] T 99663ff0fe094849acefe03db8d43b50 P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.391285 1202 ts_tablet_manager.cc:1434] T 99663ff0fe094849acefe03db8d43b50 P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251214 20:52:39.391521 1202 tablet_bootstrap.cc:492] T 47ca12fc0de44082aa3d4fabc919c1dc P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:39.393460 1202 tablet_bootstrap.cc:654] T 47ca12fc0de44082aa3d4fabc919c1dc P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
W20251214 20:52:39.401384 848 consensus_peers.cc:597] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:39.403379 1202 tablet_bootstrap.cc:492] T 47ca12fc0de44082aa3d4fabc919c1dc P a2e8fd1cb731446d88f8e510c7a52999: No bootstrap required, opened a new log
I20251214 20:52:39.403656 1202 ts_tablet_manager.cc:1403] T 47ca12fc0de44082aa3d4fabc919c1dc P a2e8fd1cb731446d88f8e510c7a52999: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20251214 20:52:39.403736 3305 raft_consensus.cc:493] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.403885 3305 raft_consensus.cc:515] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.404512 3305 leader_election.cc:290] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
W20251214 20:52:39.405475 991 leader_election.cc:343] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Not found: Tablet not found: 57a6232564194cbba12182f3a29c2ebb
I20251214 20:52:39.416445 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "57a6232564194cbba12182f3a29c2ebb" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:39.423071 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 265646e2e48a4b96984e14e659d84aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.382980 3287 raft_consensus.cc:515] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.427258 3287 leader_election.cc:290] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Requested vote from peers 63848a9fb78a4013a6f1657621272d75 (127.0.104.3:33313), 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835)
I20251214 20:52:39.427402 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7ede943c2b447709850840a9dd8b66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.428318 1202 raft_consensus.cc:359] T 47ca12fc0de44082aa3d4fabc919c1dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.428560 1202 raft_consensus.cc:385] T 47ca12fc0de44082aa3d4fabc919c1dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251214 20:52:39.428608 1202 raft_consensus.cc:740] T 47ca12fc0de44082aa3d4fabc919c1dc P a2e8fd1cb731446d88f8e510c7a52999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Initialized, Role: FOLLOWER
I20251214 20:52:39.428987 1202 consensus_queue.cc:260] T 47ca12fc0de44082aa3d4fabc919c1dc P a2e8fd1cb731446d88f8e510c7a52999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
W20251214 20:52:39.429067 989 leader_election.cc:343] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.429219 989 leader_election.cc:304] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.429446 3305 raft_consensus.cc:2749] T 57a6232564194cbba12182f3a29c2ebb P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251214 20:52:39.432065 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f629f102690489fb77e6d860067e0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.432535 3305 raft_consensus.cc:493] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.432653 3305 raft_consensus.cc:515] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.433202 3305 leader_election.cc:290] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:39.433708 910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f891db00595b49ca9e6ae64721e4d744" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:39.434185 991 leader_election.cc:343] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.435861 3532 ts_tablet_manager.cc:1916] T 292a28b12df246a8bc855d81c530746e P 63848a9fb78a4013a6f1657621272d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251214 20:52:39.435915 1206 tablet_bootstrap.cc:492] T 147c985fa7c94f7a9af57917bf17b29f P 074399864fb2470fa9e2ee87c1fe2212: Bootstrap starting.
I20251214 20:52:39.436071 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fd4e958e6e74f5ab871d40d2c31e93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251214 20:52:39.437708 1206 tablet_bootstrap.cc:654] T 147c985fa7c94f7a9af57917bf17b29f P 074399864fb2470fa9e2ee87c1fe2212: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:39.440723 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f891db00595b49ca9e6ae64721e4d744" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:39.441591 1202 ts_tablet_manager.cc:1434] T 47ca12fc0de44082aa3d4fabc919c1dc P a2e8fd1cb731446d88f8e510c7a52999: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
W20251214 20:52:39.441957 989 leader_election.cc:343] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.442126 989 leader_election.cc:304] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.442312 1202 tablet_bootstrap.cc:492] T 8c07b455b12145e28ecff35f2732b840 P a2e8fd1cb731446d88f8e510c7a52999: Bootstrap starting.
I20251214 20:52:39.442379 3305 raft_consensus.cc:2749] T f891db00595b49ca9e6ae64721e4d744 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:39.443994 1202 tablet_bootstrap.cc:654] T 8c07b455b12145e28ecff35f2732b840 P a2e8fd1cb731446d88f8e510c7a52999: Neither blocks nor log segments found. Creating new log.
I20251214 20:52:39.447839 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67104e34df294ef5986171eca79cdc72" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212"
I20251214 20:52:39.453470 962 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:39.463387 773 raft_consensus.cc:3060] T 1eff8a23d48c43e2bdbfb34f4bab6edf P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:39.463511 768 raft_consensus.cc:3060] T d171301941fa43908e05984c3b016007 P 074399864fb2470fa9e2ee87c1fe2212 [term 0 FOLLOWER]: Advancing to term 1
W20251214 20:52:39.467865 848 consensus_peers.cc:597] T 336cb9d03fd44f70930662ee3c8d987a P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251214 20:52:39.468715 848 leader_election.cc:343] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
W20251214 20:52:39.469061 848 consensus_peers.cc:597] T 4f22ac24c38e4c64bb26c54ae4d87dfb P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:39.471486 824 heartbeater.cc:499] Master 127.0.104.61:34819 was elected leader, sending a full tablet report...
I20251214 20:52:39.472645 3066 fs_manager.cc:714] Time spent opening directory manager: real 0.258s user 0.002s sys 0.007s
I20251214 20:52:39.473702 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67104e34df294ef5986171eca79cdc72" candidate_uuid: "a2e8fd1cb731446d88f8e510c7a52999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63848a9fb78a4013a6f1657621272d75"
I20251214 20:52:39.474023 1054 raft_consensus.cc:3060] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Advancing to term 1
I20251214 20:52:39.476691 1054 raft_consensus.cc:2468] T 67104e34df294ef5986171eca79cdc72 P 63848a9fb78a4013a6f1657621272d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e8fd1cb731446d88f8e510c7a52999 in term 1.
W20251214 20:52:39.480513 848 consensus_peers.cc:597] T 880a78b924574181bdd804d599f4f579 P a2e8fd1cb731446d88f8e510c7a52999 -> Peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Couldn't send request to peer 074399864fb2470fa9e2ee87c1fe2212. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251214 20:52:39.501423 3305 raft_consensus.cc:493] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251214 20:52:39.501610 3305 raft_consensus.cc:515] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38123 } }
I20251214 20:52:39.502218 3305 leader_election.cc:290] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835), a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123)
I20251214 20:52:39.502810 908 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6583412e5a1849c8b93c87ba7fdc4647" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e8fd1cb731446d88f8e510c7a52999" is_pre_election: true
W20251214 20:52:39.503360 991 leader_election.cc:343] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2e8fd1cb731446d88f8e510c7a52999 (127.0.104.2:38123): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.503638 3552 tablet.cc:1722] T 658e4fe0d188483b9ceb9eb3b3f20835 P 074399864fb2470fa9e2ee87c1fe2212: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251214 20:52:39.511672 3451 consensus_queue.cc:1048] T 0dbfe8ee68bf4a2c88489b1f81b36991 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251214 20:52:39.514896 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6583412e5a1849c8b93c87ba7fdc4647" candidate_uuid: "63848a9fb78a4013a6f1657621272d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074399864fb2470fa9e2ee87c1fe2212" is_pre_election: true
I20251214 20:52:39.517228 3553 tablet.cc:1722] T 4dcd511a5dca4a2cbda81fbc255ce354 P a2e8fd1cb731446d88f8e510c7a52999: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251214 20:52:39.519586 989 leader_election.cc:343] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 074399864fb2470fa9e2ee87c1fe2212 (127.0.104.1:36835): Illegal state: must be running to vote when last-logged opid is not known
I20251214 20:52:39.520301 989 leader_election.cc:304] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75; no voters: 074399864fb2470fa9e2ee87c1fe2212, a2e8fd1cb731446d88f8e510c7a52999
I20251214 20:52:39.520550 3371 raft_consensus.cc:2749] T 6583412e5a1849c8b93c87ba7fdc4647 P 63848a9fb78a4013a6f1657621272d75 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251214 20:52:39.532938 851 leader_election.cc:304] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63848a9fb78a4013a6f1657621272d75, a2e8fd1cb731446d88f8e510c7a52999; no voters: 074399864fb2470fa9e2ee87c1fe2212
I20251214 20:52:39.535948 3287 raft_consensus.cc:2804] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 FOLLOWER]: Leader election won for term 1
I20251214 20:52:39.536130 3287 raft_consensus.cc:697] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [term 1 LEADER]: Becoming Leader. State: Replica: a2e8fd1cb731446d88f8e510c7a52999, State: Running, Role: LEADER
I20251214 20:52:39.536617 3287 consensus_queue.cc:237] T 67104e34df294ef5986171eca79cdc72 P a2e8fd1cb731446d88f8e510c7a52999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63848a9fb78a4013a6f1657621272d75" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33313 } } peers { permanent_uuid: "074399864fb2470fa9e2ee87c1fe2212" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36835 } } peers { permanent_uuid: "a2e8fd1cb731446d88f8e510c7a52999" member_type: VOTER last_known_addr { host: "127.0.104.2"